From 34071ac66087411f9888bf79587de62d7225faa3 Mon Sep 17 00:00:00 2001 From: Zakhar Bessarab Date: Wed, 12 Jun 2024 18:34:18 +0400 Subject: [PATCH] lib/promscrape: increase default value for promscrape.maxDroppedTargets to 10_000 (#6459) ### Describe Your Changes This limit can be increased since after https://github.com/VictoriaMetrics/VictoriaMetrics/commit/4513893ead4ecb4a6dd6b8d0ed7b2bd6bc0cafe5 tracking of dropped targets uses much less memory per entry. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381#issuecomment-2156708228 ### Checklist The following checks are **mandatory**: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Zakhar Bessarab --- README.md | 2 +- docs/CHANGELOG.md | 1 + docs/README.md | 2 +- docs/Single-server-VictoriaMetrics.md | 2 +- docs/vmagent.md | 2 +- lib/promscrape/targetstatus.go | 7 ++++--- 6 files changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 1cb330ddd..d26c5ede4 100644 --- a/README.md +++ b/README.md @@ -3027,7 +3027,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -promscrape.kumaSDCheckInterval duration Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int - The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 1000) + The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size The maximum size of http response headers from Prometheus scrape targets Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 4096) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index bb8bdf0e2..1276c5ce2 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -36,6 +36,7 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/). * FEATURE: [dashboards/vmagent](https://grafana.com/grafana/dashboards/12683): add row `Streaming aggregation` with panels related to [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/) process. * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add `idleConnTimeout` flag set to 50s by default. It should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmauth logs. * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add auto request retry for trivial network errors, such as `broken pipe` and `connection reset` for requests to the configured backends. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): increase default value of `-promscrape.maxDroppedTargets` command-line flag to 10_000 from 1000. This makes it easier to track down large number of dropped targets. * BUGFIX: all VictoriaMetrics components: prioritize `-configAuthKey` and `-reloadAuthKey` over `-httpAuth.*` settings. This change aligns behavior of mentioned flags with other auth flags like `-metricsAuthKey`, `-flagsAuthKey`, `-pprofAuthKey`. Check [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329). * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--disable-progress-bar` global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. `--vm-disable-progress-bar` command-line flag is deprecated and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367). diff --git a/docs/README.md b/docs/README.md index 799eeea8b..cc743edbd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3030,7 +3030,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -promscrape.kumaSDCheckInterval duration Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int - The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 1000) + The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size The maximum size of http response headers from Prometheus scrape targets Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 4096) diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index ed4c30afe..0f6e198b8 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -3038,7 +3038,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -promscrape.kumaSDCheckInterval duration Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int - The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 1000) + The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size The maximum size of http response headers from Prometheus scrape targets Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 4096) diff --git a/docs/vmagent.md b/docs/vmagent.md index 872514117..85bc603cc 100644 --- a/docs/vmagent.md +++ b/docs/vmagent.md @@ -1955,7 +1955,7 @@ See the docs at https://docs.victoriametrics.com/vmagent/ . -promscrape.kumaSDCheckInterval duration Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int - The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 1000) + The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size The maximum size of http response headers from Prometheus scrape targets Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 4096) diff --git a/lib/promscrape/targetstatus.go b/lib/promscrape/targetstatus.go index b0ec49542..afdd47d64 100644 --- a/lib/promscrape/targetstatus.go +++ b/lib/promscrape/targetstatus.go @@ -13,14 +13,15 @@ import ( "time" "unsafe" + "github.com/VictoriaMetrics/metrics" + "github.com/cespare/xxhash/v2" + "github.com/VictoriaMetrics/VictoriaMetrics/lib/logger" "github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel" "github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils" - "github.com/VictoriaMetrics/metrics" - "github.com/cespare/xxhash/v2" ) -var maxDroppedTargets = flag.Int("promscrape.maxDroppedTargets", 1000, "The maximum number of droppedTargets to show at /api/v1/targets page. "+ +var maxDroppedTargets = flag.Int("promscrape.maxDroppedTargets", 10000, "The maximum number of droppedTargets to show at /api/v1/targets page. "+ "Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. "+ "Note that the increased number of tracked dropped targets may result in increased memory usage")