Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 16:30:55 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
09c388a8e4
VictoriaMetrics/lib
History
Aliaksandr Valialkin 09c388a8e4
lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode 
While fasthttp.Client uses less CPU and RAM when scraping targets with small responses (up to 10K metrics),
it doesn't work well when scraping targets with big responses such as kube-state-metrics.
In this case it could use big amounts of additional memory comparing to net/http.Client,
since fasthttp.Client reads the full response in memory and then tries re-using the large buffer
for further scrapes.

Additionally, fasthttp.Client-based scraping had various issues with proxying, redirects
and scrape timeouts like the following ones:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5425
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017

This should help reducing memory usage for the case when target returns big response
and this response is scraped by fasthttp.Client at first before switching to stream parsing mode
for subsequent scrapes. Now the switch to stream parsing mode is performed on the first scrape
after reading the response body in memory and noticing that its size exceeds the value passed
to -promscrape.minResponseSizeForStreamParse command-line flag.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567

Overrides https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931
2024-01-30 18:39:55 +02:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:26:02 +02:00
auth lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant 2023-08-30 14:13:51 +02:00
awsapi lib/awsapi: properly assume role with webIdentity token (#5495) 2023-12-20 19:07:04 +02:00
backup vendor: run make vendor-update 2023-12-11 10:48:47 +02:00
blockcache all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
bloomfilter all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time 2024-01-22 01:13:41 +02:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-07-06 17:22:23 -07:00
buildinfo
bytesutil lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args 2023-11-13 09:43:49 +01:00
cgroup lib/cgroup: add SetGOGC() function 2023-07-06 17:24:31 -07:00
consts
decimal lib/decimal: follow-up for e6bad5174f 2024-01-27 00:08:32 +01:00
encoding lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint* 2024-01-23 22:50:11 +02:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:37:05 +02:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 17:57:19 -07:00
fastnum
fasttime
filestream lib/filestream: do not measure read / write duration from / to in-memory buffers 2024-01-23 14:53:35 +02:00
flagutil add inserting comma inside value instruction to flag description (#5666) 2024-01-26 22:47:33 +01:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:43 -08:00
fs all: allow dynamically reading *AuthKey flag values from files and urls 2024-01-22 01:23:23 +02:00
handshake lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow 2024-01-23 18:39:28 +02:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:28:17 +01:00
httpserver all: allow dynamically reading *AuthKey flag values from files and urls 2024-01-22 01:23:23 +02:00
httputils lib/httputils: handle step=undefined query arg as an empty value 2024-01-17 00:13:04 +02:00
influxutils lib/flagutil: rename Array to ArrayString 2022-10-01 18:28:19 +03:00
ingestserver lib/ingestserver: properly log the number of closed connections 2023-11-14 21:53:10 +01:00
leveledbytebufferpool all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
logger lib/logger: increase default -loggerMaxArgLen command-line flag value from 500 to 1000 2023-11-14 19:55:55 +01:00
logjson app/vlinsert/jsonline: code prettifying 2023-07-06 21:35:55 -07:00
logstorage lib/logstorage: make sure that WaitGroup.Add isnt called after stopCh is closed and WaitGroup.Wait is called 2024-01-26 21:18:07 +01:00
lrucache all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
mergeset lib/{mergeset,storage}: make background merge more responsive and scalable 2024-01-26 22:19:52 +01:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
netutil app/vmselect: add support for vmstorage groups with independent -replicationFactor per group 2023-12-13 00:14:34 +02:00
persistentqueue app/vmagent: follow-up for 090cb2c9de 2023-11-25 12:13:39 +02:00
procutil all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
promauth lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode 2024-01-30 18:39:55 +02:00
prompb lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto 2024-01-16 20:48:30 +02:00
prompbmarshal lib/prompbmarshal: move WriteRequest proto definition to the correct place 2024-01-16 21:57:03 +02:00
promrelabel all: allow dynamically reading *AuthKey flag values from files and urls 2024-01-22 01:23:23 +02:00
promscrape lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode 2024-01-30 18:39:55 +02:00
promutils all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually 2024-01-22 01:11:59 +02:00
protoparser all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually 2024-01-22 01:11:59 +02:00
proxy lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode 2024-01-30 18:39:55 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-16 21:18:22 +02:00
querytracer lib/querytracer: add missing blank comment line after 3121d76bee 2023-11-15 16:11:50 +01:00
regexutil lib/regexutil: properly handle alternate regexps surrounded by .+ or .* 2023-11-13 18:25:57 +01:00
snapshot Makefile: update golangci-lint version from v1.54.2 to v1.55.1 2023-11-02 21:42:35 +01:00
storage lib/storage: keep (date, metricID) entries only for the last two dates 2024-01-29 18:44:27 +01:00
streamaggr lib/streamaggr: fix incorrect err message for min interval value 2024-01-29 17:27:23 +01:00
stringsutil lib/stringsutil: add tests for LimitStringLen() function 2023-11-13 10:33:07 +01:00
syncwg
tenantmetrics lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 2023-07-20 21:54:51 -07:00
timerpool
timeutil all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
uint64set lib/uint64: remove accidentally added test 2024-01-09 13:32:22 +01:00
vmselectapi vmcluster: re-routing enhancement (#5293) 2023-11-14 01:00:42 +01:00
workingsetcache all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
writeconcurrencylimiter lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting vm_concurrent_insert_capacity and vm_concurrent_insert_current metrics 2023-02-07 11:08:39 -08:00