From c0039ce7a3506918de538eb30edf958c79a5cd65 Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Thu, 16 Nov 2023 16:26:57 +0100 Subject: [PATCH] docs/vmalert: clarify deduplication recommendations for HA setup (#5336) Please see discussion here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5279 Signed-off-by: hagen1778 --- app/vmalert/README.md | 14 ++++++++++++-- docs/vmalert.md | 14 ++++++++++++-- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/app/vmalert/README.md b/app/vmalert/README.md index b47b5fc239..0de72579a0 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -542,10 +542,20 @@ Alertmanagers. To avoid recording rules results and alerts state duplication in VictoriaMetrics server don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). -The recommended value for `-dedup.minScrapeInterval` must be multiple of vmalert's `-evaluationInterval`. +Multiple equally configured vmalerts should evaluate rules at the same timestamps, so it is recommended +to set `-dedup.minScrapeInterval` as equal to vmalert's `-evaluationInterval`. + +If you have multiple different `interval` params for distinct rule groups, then set `-dedup.minScrapeInterval` to +the biggest `interval` value, or value which will be a multiple for all `interval` values. For example, if you have +two groups with `interval: 10s` and `interval: 15s`, then set `-dedup.minScrapeInterval=30s`. This would consistently +keep only a single data point on 30s time interval for all rules. However, try to avoid having inconsistent `interval` +values. + +It is not recommended having `-dedup.minScrapeInterval` smaller than `-evaluationInterval`, as it may produce +results with inconsistent intervals between data points. Alertmanager will automatically deduplicate alerts with identical labels, so ensure that -all `vmalert`s are having the same config. +all `vmalert`s are having identical config. Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/) for Alertmanagers for better reliability. List all Alertmanager URLs in vmalert `-notifier.url` diff --git a/docs/vmalert.md b/docs/vmalert.md index 8605e37d44..63c3916418 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -553,10 +553,20 @@ Alertmanagers. To avoid recording rules results and alerts state duplication in VictoriaMetrics server don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). -The recommended value for `-dedup.minScrapeInterval` must be multiple of vmalert's `-evaluationInterval`. +Multiple equally configured vmalerts should evaluate rules at the same timestamps, so it is recommended +to set `-dedup.minScrapeInterval` as equal to vmalert's `-evaluationInterval`. + +If you have multiple different `interval` params for distinct rule groups, then set `-dedup.minScrapeInterval` to +the biggest `interval` value, or value which will be a multiple for all `interval` values. For example, if you have +two groups with `interval: 10s` and `interval: 15s`, then set `-dedup.minScrapeInterval=30s`. This would consistently +keep only a single data point on 30s time interval for all rules. However, try to avoid having inconsistent `interval` +values. + +It is not recommended having `-dedup.minScrapeInterval` smaller than `-evaluationInterval`, as it may produce +results with inconsistent intervals between data points. Alertmanager will automatically deduplicate alerts with identical labels, so ensure that -all `vmalert`s are having the same config. +all `vmalert`s are having identical config. Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/) for Alertmanagers for better reliability. List all Alertmanager URLs in vmalert `-notifier.url`