VictoriaMetrics/app/vmalert
Roman Khavronenko 562edb72ea
app/vmalert: fix data race during hot-config reload (#5698)
* app/vmalert: fix data race during hot-config reload

During hot-reload, the logic evokes the group update and rules evaluation
interruption simultaneously. Falsely assuming that interruption happens before
the update. However, it could happen that group will be updated first and only
after the rules evaluation will be cancelled. Which will result in permanent
interruption for all rules within the group.

The fix caches the cancel context function into local variable first. And only after
performs the group update. With cached cancel function we can safely call it without
worrying that we cancel the evaluation for already updated group.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "app/vmalert: fix data race during hot-config reload"

This reverts commit a4bb7e8932.

* app/vmalert: fix data race during hot-config reload

During hot-reload, the logic evokes the group update and rules evaluation
interruption simultaneously. Falsely assuming that interruption happens before
the update. However, it could happen that group will be updated first and only
after the rules evaluation will be cancelled. Which will result in permanent
interruption for all rules within the group.

The fix cancels the evaulation context before applying the update, making sure
that the context will be cancelled for old group always.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 22:43:02 +01:00
..
config docs: fix Grafana link example for vmalert 2024-01-22 18:41:38 +02:00
datasource app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags 2023-12-20 21:38:16 +02:00
deployment app/vmalert: include it into the next release 2020-04-28 00:11:41 +03:00
multiarch all: run apk update && apk upgrade in base Alpine Docker image in order to get all the recent security fixes 2023-02-09 14:03:02 -08:00
notifier app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags 2023-12-20 21:38:16 +02:00
remoteread app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags 2023-12-20 21:38:16 +02:00
remotewrite lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto 2024-01-16 20:48:30 +02:00
rule app/vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0 (#5680) 2024-01-26 20:51:50 +01:00
static app/vmalert: show on UI groups error after reload config (#4543) 2023-07-06 22:11:36 -07:00
templates app/vmalert: add support of recursive path globs for rules and templates (#4148) 2023-05-08 16:22:30 -07:00
tpl app/vmalert: show on UI groups error after reload config (#4543) 2023-07-06 22:11:36 -07:00
utils app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags 2023-12-20 21:38:16 +02:00
main_test.go all: use Gauge instead of Counter for *_config_last_reload_successful metrics 2023-12-20 14:25:44 +02:00
main.go lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-16 21:18:22 +02:00
Makefile app/vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0 (#5680) 2024-01-26 20:51:50 +01:00
manager_test.go vmalert-tool: implement unittest (#4789) 2023-10-16 14:12:06 +02:00
manager.go app/vmalert: fix data race during hot-config reload (#5698) 2024-01-26 22:43:02 +01:00
README.md docs: convert png images to webp in all the docs except of docs/operator/* 2023-11-22 19:29:47 +02:00
replay_test.go vmalert-tool: implement unittest (#4789) 2023-10-16 14:12:06 +02:00
replay.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-26 09:44:40 +02:00
web_test.go app/vmalert: expose /vmalert/api/v1/rule and /api/v1/rule API which returns rule status in JSON format (#5397) 2023-12-04 22:49:39 +02:00
web_types_test.go vmalert-tool: implement unittest (#4789) 2023-10-16 14:12:06 +02:00
web_types.go app/vmalert: expose /vmalert/api/v1/rule and /api/v1/rule API which returns rule status in JSON format (#5397) 2023-12-04 22:49:39 +02:00
web.go all: allow dynamically reading *AuthKey flag values from files and urls 2024-01-22 01:23:23 +02:00
web.qtpl vmalert-tool: implement unittest (#4789) 2023-10-16 14:12:06 +02:00
web.qtpl.go vmalert-tool: implement unittest (#4789) 2023-10-16 14:12:06 +02:00

See vmalert docs here.

vmalert docs can be edited at docs/vmalert.md.