65 Commits

Author	SHA1	Message	Date
Zakhar Bessarab	de5aad2cde	lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568)	2023-01-05 21:23:15 -08:00
Aliaksandr Valialkin	d3f8298739	lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..))	2023-01-03 22:15:49 -08:00
Aliaksandr Valialkin	8e9548f050	lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize ... This is a follow-up for 7ad9fff7e5	2022-12-28 14:42:45 -08:00
Clément Nussbaumer	04d536c15a	fix(promscrape): check MaxScrapeSize after gzip decompression (#3550)	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	1a88fe5b1f	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures ... This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:29:57 -08:00
Aliaksandr Valialkin	be6da5053f	lib/promscrape: optimize service discovery speed ... - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	5794886662	lib/promscrape: properly set Host header when sending requests via http proxy ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets ... This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() ... Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit 67b10896d2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530) ... standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Aliaksandr Valialkin	5bd4e47a9e	lib/promscrape: allow up to 5 redirects when scraping a target by default ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:45 +02:00
Aliaksandr Valialkin	d445d22c0c	lib/promscrape: add -promscrape.maxResponseHeadersSize command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:27:55 +02:00
Aliaksandr Valialkin	ad445a06cd	lib/promscrape: properly show proxy_url option value at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin	99011c6b63	lib/promscrape: always initialize http client for stream parsing mode ... Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:19:48 +03:00
Aliaksandr Valialkin	d8de26bbfd	lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin	d90834da70	lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does ... See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin	df057177a0	lib/promscrape: increase the duration for reading the full response in stream parsing mode ... Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin	97de72054e	docs: document f0c21b6300	2021-05-27 15:04:13 +03:00
faceair	b801b299f0	lib/promscrape: apply body size & sample limit to stream parse (#1331) ... * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 15:04:11 +03:00
Aliaksandr Valialkin	e08287f017	lib/promscrape: reload auth tokens from files every second ... Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:03:35 +03:00
Aliaksandr Valialkin	a6cb4f10a7	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport ... By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin	f13585dc5d	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:09 +03:00
Aliaksandr Valialkin	d13906bf1f	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services ... This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:07 +03:00
Aliaksandr Valialkin	320983f650	lib/promscrape: apply scrape_timeout on receiving the first response byte for stream_parse: true scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047	2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin	90da332ea0	lib/promscrape: pass X-Prometheus-Scrape-Timeout-Seconds header to scrape targets as Prometheus does	2021-04-05 12:15:14 +03:00
Aliaksandr Valialkin	dd19fab7c9	lib/promscrape: properly send full url in GET request via simple HTTP proxy ... This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 01:20:43 +03:00
Aliaksandr Valialkin	ab9e1eb41f	lib/promscrape: support for simple HTTP proxies without CONNECT method support such as https://github.com/prometheus-community/PushProx ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 00:40:58 +03:00
Aliaksandr Valialkin	2825a1e86d	lib/promscrape: add follow_redirect option to scrape_configs section like Prometheus does ... See https://github.com/prometheus/prometheus/pull/8546	2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin	e2717d84c0	all: various fixes in command-line flag descriptions	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	60e0280a94	lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 03:36:11 +02:00
Aliaksandr Valialkin	9d8223eafb	lib/proxy: set missing ServerName in TLS config for proxy_url. ... While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.	2021-03-09 19:01:14 +02:00
Nikolay	1310f84122	Changes tlsConfig init for proxy connections (#1121) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-09 19:01:13 +02:00
Aliaksandr Valialkin	4c59dbc127	lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics	2021-02-01 20:06:16 +02:00
Aliaksandr Valialkin	fdced59278	lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection	2021-01-22 13:22:59 +02:00
Aliaksandr Valialkin	367fc17933	lib/promscrape: code prettifying for 8dd03ecf19	2020-12-24 10:57:20 +02:00
Nikolay	b00f7816e2	adds proxy_url support, (#980) ... * adds proxy_url support, adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery, adds proxy_url to the scrape_config for targets scrapping, http based proxy is supported atm, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503 * fixes imports	2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin	3bb9bf33d6	lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets	2020-11-26 18:32:16 +02:00
Aliaksandr Valialkin	3dd2282ed9	lib/promscrape: release http response non-200 status code	2020-11-26 13:25:25 +02:00
Aliaksandr Valialkin	3ed9f1d5a9	lib/promscrape: properly handle response body after 301 redirect ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:59 +02:00
Nikolay	e8fe618bbb	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-11-01 23:25:23 +02:00
Aliaksandr Valialkin	ed724d25ba	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin	8fd791d399	lib/promscrape: avoid copying response body when scraping targets. ... This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:06:00 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	1724cc241e	lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes	2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin	87f8c728bf	lib/promscrape: send Accept header similar to Prometheus when scraping targets ... This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:50:06 +03:00
Aliaksandr Valialkin	6f1d926698	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:48 +03:00
Aliaksandr Valialkin	ee03b4ccbd	lib/promscrape: prevent from too big deadline misses on scrape retries ... The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:42:09 +03:00
Aliaksandr Valialkin	dfa83a4a35	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:25 +03:00