63 Commits

Author	SHA1	Message	Date
Zakhar Bessarab	bced9fb978	lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568)	2023-01-05 19:34:47 -08:00
Aliaksandr Valialkin	add2c4bf07	lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..))	2023-01-03 22:16:22 -08:00
Aliaksandr Valialkin	c4229a1bba	lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize ... This is a follow-up for 7ad9fff7e5	2022-12-28 14:42:11 -08:00
Clément Nussbaumer	7ad9fff7e5	fix(promscrape): check MaxScrapeSize after gzip decompression (#3550)	2022-12-28 12:19:41 -08:00
Aliaksandr Valialkin	4de9d35458	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures ... This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin	f325410c26	lib/promscrape: optimize service discovery speed ... - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin	1f89278d88	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin	bb68ab99fa	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin	f97355d9fb	lib/promscrape: properly set Host header when sending requests via http proxy ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin	1c4f67c5d2	lib/promauth: add ability to send additional http headers in requests to scrape targets ... This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin	123aa4c79e	lib/promscrape: properly implement ScrapeConfig.clone() ... Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit 67b10896d2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:05:40 +03:00
Nikolay	3575aabeaf	lib/promscrape: adds correct http status codes for redirect (#2530) ... standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 13:31:31 +03:00
Aliaksandr Valialkin	a3adf24527	lib/promscrape: allow up to 5 redirects when scraping a target by default ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:14 +02:00
Aliaksandr Valialkin	aa534c2582	lib/promscrape: add -promscrape.maxResponseHeadersSize command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:26:56 +02:00
Aliaksandr Valialkin	4821adfd95	lib/promscrape: properly show proxy_url option value at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:23:54 +03:00
Aliaksandr Valialkin	06b0982d6b	lib/promscrape: always initialize http client for stream parsing mode ... Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:18:23 +03:00
Aliaksandr Valialkin	a0313c046b	lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:47:54 +03:00
Aliaksandr Valialkin	728c4c3841	lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does ... See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:20:44 +03:00
Aliaksandr Valialkin	06b8e7d148	lib/promscrape: increase the duration for reading the full response in stream parsing mode ... Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:28:09 +03:00
Aliaksandr Valialkin	60341722d5	docs: document f0c21b6300	2021-05-27 15:03:30 +03:00
faceair	f0c21b6300	lib/promscrape: apply body size & sample limit to stream parse (#1331) ... * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 14:52:44 +03:00
Aliaksandr Valialkin	733706e6c6	lib/promscrape: reload auth tokens from files every second ... Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:00:08 +03:00
Aliaksandr Valialkin	10a47af631	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport ... By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:12:24 +03:00
Aliaksandr Valialkin	e6fda03e8f	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:44:14 +03:00
Aliaksandr Valialkin	f6a641de62	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services ... This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:38:50 +03:00
Aliaksandr Valialkin	908e35affd	lib/promscrape: apply scrape_timeout on receiving the first response byte for stream_parse: true scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047	2021-04-23 21:53:35 +03:00
Aliaksandr Valialkin	6742839fd6	lib/promscrape: pass X-Prometheus-Scrape-Timeout-Seconds header to scrape targets as Prometheus does	2021-04-05 12:15:24 +03:00
Aliaksandr Valialkin	500e625e8c	lib/promscrape: properly send full url in GET request via simple HTTP proxy ... This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 01:20:06 +03:00
Aliaksandr Valialkin	5153410ced	lib/promscrape: support for simple HTTP proxies without CONNECT method support such as https://github.com/prometheus-community/PushProx ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 00:40:40 +03:00
Aliaksandr Valialkin	7f9c68cdcb	lib/promscrape: add follow_redirect option to scrape_configs section like Prometheus does ... See https://github.com/prometheus/prometheus/pull/8546	2021-04-02 19:56:40 +03:00
Aliaksandr Valialkin	85a95bf60c	all: various fixes in command-line flag descriptions	2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin	a6a71ef861	lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 03:36:19 +02:00
Aliaksandr Valialkin	36fd007247	lib/proxy: set missing ServerName in TLS config for proxy_url. ... While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.	2021-03-09 18:58:18 +02:00
Nikolay	ad34f42467	Changes tlsConfig init for proxy connections (#1121) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-09 18:51:00 +02:00
Aliaksandr Valialkin	bd11fd8f1d	lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics	2021-02-01 20:06:27 +02:00
Aliaksandr Valialkin	8cea3c3cc4	lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection	2021-01-22 13:22:33 +02:00
Aliaksandr Valialkin	820669da69	lib/promscrape: code prettifying for 8dd03ecf19	2020-12-24 10:56:10 +02:00
Nikolay	8dd03ecf19	adds proxy_url support, (#980) ... * adds proxy_url support, adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery, adds proxy_url to the scrape_config for targets scrapping, http based proxy is supported atm, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503 * fixes imports	2020-12-24 10:52:37 +02:00
Aliaksandr Valialkin	3a32789352	lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets	2020-11-26 18:32:06 +02:00
Aliaksandr Valialkin	b0a5c382ee	lib/promscrape: release http response non-200 status code	2020-11-26 13:25:17 +02:00
Aliaksandr Valialkin	c0f3be824d	lib/promscrape: properly handle response body after 301 redirect ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:52 +02:00
Aliaksandr Valialkin	e277c3d07b	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:35:06 +02:00
Nikolay	67059caa12	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-10-29 17:17:52 +03:00
Aliaksandr Valialkin	1a9ee39b0e	lib/promscrape: avoid copying response body when scraping targets. ... This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:05:43 +03:00
Aliaksandr Valialkin	147c35ebd4	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin	eead3ee8ec	lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes	2020-08-13 22:31:52 +03:00
Aliaksandr Valialkin	3e1d7d8489	lib/promscrape: send Accept header similar to Prometheus when scraping targets ... This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:48:22 +03:00
Aliaksandr Valialkin	ff6a0955eb	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:22 +03:00
Aliaksandr Valialkin	8b133e40d5	lib/promscrape: prevent from too big deadline misses on scrape retries ... The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:41:36 +03:00
Aliaksandr Valialkin	44a54b8b3d	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:14 +03:00