diff --git a/app/vmalert/README.md b/app/vmalert/README.md index e37e04bce..b6fecdc74 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -2,7 +2,11 @@ `vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) -rules against configured address. It is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/) +rules against configured `-datasource.url`. For sending alerting notifications +vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag. +Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) +protocol and require `-remoteWrite.url` to be configured. +Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/) implementation and aims to be compatible with its syntax. ## Features @@ -18,12 +22,12 @@ implementation and aims to be compatible with its syntax. * Lightweight without extra dependencies. ## Limitations -* `vmalert` execute queries against remote datasource which has reliability risks because of network. -It is recommended to configure alerts thresholds and rules expressions with understanding that network request -may fail; +* `vmalert` execute queries against remote datasource which has reliability risks because of the network. +It is recommended to configure alerts thresholds and rules expressions with the understanding that network +requests may fail; * by default, rules execution is sequential within one group, but persistence of execution results to remote storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous -recording rule is reused in next one; +recording rule is reused in the next one; ## QuickStart @@ -33,13 +37,13 @@ git clone https://github.com/VictoriaMetrics/VictoriaMetrics cd VictoriaMetrics make vmalert ``` -The build binary will be placed to `VictoriaMetrics/bin` folder. +The build binary will be placed in `VictoriaMetrics/bin` folder. To start using `vmalert` you will need the following things: * list of rules - PromQL/MetricsQL expressions to execute; * datasource address - reachable MetricsQL endpoint to run queries against; * notifier address [optional] - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing, -aggregating alerts and sending notifications. +aggregating alerts, and sending notifications. * remote write address [optional] - [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) compatible storage to persist rules and alerts state info; * remote read address [optional] - MetricsQL compatible datasource to restore alerts state from. @@ -58,7 +62,7 @@ Then configure `vmalert` accordingly: Note there's a separate `remoteRead.url` to allow writing results of alerting/recording rules into a different storage than the initial data that's -queried. This allows using `vmalert` to aggregate data from a short-term, +queried. This allows using `vmalert` to aggregate data from a short-term, high-frequency, high-cardinality storage into a long-term storage with decreased cardinality and a bigger interval between samples. @@ -120,14 +124,14 @@ expression and then act according to the Rule type. There are two types of Rules: * [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - -Alerting rules allow to define alert conditions via `expr` field and to send notifications to +Alerting rules allow defining alert conditions via `expr` field and to send notifications to [Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty. * [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) - -Recording rules allow to define `expr` which result will be then backfilled to configured +Recording rules allow defining `expr` which result will be then backfilled to configured `-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. -`vmalert` forbids defining duplicates - rules with the same combination of name, expression and labels +`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels within one group. #### Alerting rules @@ -143,7 +147,7 @@ alert: expr: # Alerts are considered firing once they have been returned for this long. -# Alerts which have not yet fired for long enough are considered pending. +# Alerts which have not yet been fired for long enough are considered pending. # If param is omitted or set to 0 then alerts will be immediately considered # as firing once they return. [ for: | default = 0s ] @@ -188,19 +192,19 @@ For recording rules to work `-remoteWrite.url` must be specified. the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. -These are regular time series and may be queried from VM just as any other time series. +These are regular time series and maybe queried from VM just as any other time series. The state is stored to the configured address on every rule evaluation. * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state from configured address by querying time series with name `ALERTS_FOR_STATE`. -Both flags are required for proper state restoring. Restore process may fail if time series are missing +Both flags are required for proper state restoration. Restore process may fail if time series are missing in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) or received state doesn't match current `vmalert` rules configuration. ### Multitenancy -The are the following approaches exist for alerting and recording rules across +There are the following approaches exist for alerting and recording rules across [multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy): * To run a separate `vmalert` instance per each tenant. @@ -240,6 +244,106 @@ The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags). +### Topology examples + +The following sections are showing how `vmalert` may be used and configured +for different scenarios. + +Please note, not all flags in examples are required: +* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if +you have recording rules or want to store [alerts state](#alerts-state-on-restarts) on `vmalert` restarts; +* `-notifier.url` is optional and is needed only if you have alerting rules. + +#### Single-node VictoriaMetrics + +vmalert single + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions + -remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results + -remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart + -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger +``` + +The simplest configuration where one single-node VM server is used for +rules execution, storing recording rules results and alerts state. + +#### Cluster VictoriaMetrics + +vmalert cluster + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions + -remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results + -remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart + -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger +``` + +In [cluster mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) +VictoriaMetrics has separate components for writing and reading path: +`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions +and `vminsert` is used to persist recording rules results and alerts state. +Cluster mode could have multiple `vminsert` and `vmselect` components. In case when you want +to spread the load on these components - add balancers before them and configure +`vmalert` with balancer's addresses. Please, see more about VM's cluster architecture +[here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview). + +#### vmalert high availability + +vmalert ha + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions + -remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results + -remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart + -notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger + -notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers +``` + +For HA user can run multiple identically configured `vmalert` instances. +It means all of them will execute the same rules, write state and results to +the same destinations, and send alert notifications to multiple configured +Alertmanagers. + +To avoid recording rules results and alerts state duplication in VictoriaMetrics server +don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). + +Alertmanager will automatically deduplicate alerts with identical labels, so ensure that +all `vmalert`s are having the same config. + +Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/) +for Alertmanagers for better reliability. + +This example uses single-node VM server for the sake of simplicity. +Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics) if needed. + +#### Downsampling and aggregation via vmalert + +vmalert multi cluster + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions + -remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results +``` + +Example shows how to build a topology where `vmalert` will process data from one cluster +and write results into another. Such clusters may be called as "hot" (low retention, +high-speed disks, used for operative monitoring) and "cold" (long term retention, +slower/cheaper disks, low resolution data). With help of `vmalert`, user can setup +recording rules to process raw data from "hot" cluster (by applying additional transformations +or reducing resolution) and push results to "cold" cluster. + +Please note, [replay](#rules-backfilling) feature may be used for transforming historical data. + +Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used. ### Web @@ -259,7 +363,7 @@ vmalert sends requests to `<-datasource.url>/render?format=json` during evaluati if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render` implements [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) for `format=json`. When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget -to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on query type. +to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on the query type. ## Rules backfilling @@ -318,11 +422,11 @@ to prevent cache pollution and unwanted time range boundaries adjustment during #### Recording rules -Result of recording rules `replay` should match with results of normal rules evaluation. +The result of recording rules `replay` should match with results of normal rules evaluation. #### Alerting rules -Result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts). +The result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts). To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression: ``` ALERTS{alertname="your_alertname", alertstate="firing"} @@ -335,7 +439,7 @@ There are following non-required `replay` flags: * `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request. In two words, it affects the max time range for every `/query_range` request. The higher the value, -the less requests will be issued during `replay`. +the fewer requests will be issued during `replay`. * `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries per rule before giving up. * `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining @@ -357,7 +461,7 @@ See full description for these flags in `./vmalert --help`. We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported metrics may be analyzed later. -Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. +Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. If you have suggestions for improvements or have found a bug - please open an issue on github or add a review to the dashboard. diff --git a/app/vmalert/vmalert_cluster.excalidraw b/app/vmalert/vmalert_cluster.excalidraw new file mode 100644 index 000000000..e256b5ea4 --- /dev/null +++ b/app/vmalert/vmalert_cluster.excalidraw @@ -0,0 +1,525 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "type": "rectangle", + "version": 794, + "versionNonce": 1855937036, + "isDeleted": false, + "id": "VgBUzo0blGR-Ijd2mQEEf", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 422.3502197265625, + "y": 215.55953979492188, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 123.7601318359375, + "height": 72.13211059570312, + "seed": 1194011660, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "miEbzHxOPXe4PEYvXiJp5", + "rcmiQfIWtfbTTlwxqr1sl", + "P-dpWlSTtnsux-zr5oqgF", + "oAToSPttH7aWoD_AqXGFX", + "Bpy5by47XGKB4yS99ZkuA", + "wRO0q9xKPHc8e8XPPsQWh", + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348083348 + }, + { + "type": "text", + "version": 659, + "versionNonce": 247957684, + "isDeleted": false, + "id": "e9TDm09y-GhPm84XWt0Jv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 443.89678955078125, + "y": 236.64378356933594, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 82, + "height": 24, + "seed": 327273100, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347948032, + "fontSize": 20, + "fontFamily": 3, + "text": "vmalert", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "middle" + }, + { + "type": "rectangle", + "version": 1670, + "versionNonce": 2021681972, + "isDeleted": false, + "id": "dd52BjHfPMPRji9Tws7U-", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 750.3317260742188, + "y": 226.5509033203125, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 1779959692, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "Bpy5by47XGKB4yS99ZkuA" + ], + "updated": 1638348054411 + }, + { + "type": "text", + "version": 1311, + "versionNonce": 1283453068, + "isDeleted": false, + "id": "9TEzv0sVCHAkc46ou0oNF", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 759.2862243652344, + "y": 238.68240356445312, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 152, + "height": 24, + "seed": 1617178804, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348054411, + "fontSize": 20, + "fontFamily": 3, + "text": "vminsert:8480", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 897, + "versionNonce": 1983434892, + "isDeleted": false, + "id": "Sa4OBd1ZjD6itohm7Ll8z", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 61.744873046875, + "y": 224.9600830078125, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 126267060, + "groupIds": [ + "ek-pq3umtz1yN-J_-preq" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh" + ], + "updated": 1638348077724 + }, + { + "type": "text", + "version": 719, + "versionNonce": 457402292, + "isDeleted": false, + "id": "we766A079lfGYu2_aC4Pl", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 70.69937133789062, + "y": 237.33523559570312, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 152, + "height": 24, + "seed": 478660236, + "groupIds": [ + "ek-pq3umtz1yN-J_-preq" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348077725, + "fontSize": 20, + "fontFamily": 3, + "text": "vmselect:8481", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 1098, + "versionNonce": 1480603788, + "isDeleted": false, + "id": "8-XFSbd6Zw96EUSJbJXZv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 371.7434387207031, + "y": 398.50787353515625, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 240.10644531249997, + "height": 44.74725341796875, + "seed": 99322124, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348083348 + }, + { + "type": "text", + "version": 864, + "versionNonce": 1115813900, + "isDeleted": false, + "id": "GUs816aggGqUSdoEsSmea", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 393.73809814453125, + "y": 410.5976257324219, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 199, + "height": 24, + "seed": 1194745268, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348009180, + "fontSize": 20, + "fontFamily": 3, + "text": "alertmanager:9093", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "arrow", + "version": 2405, + "versionNonce": 959767732, + "isDeleted": false, + "id": "Bpy5by47XGKB4yS99ZkuA", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 556.6860961914062, + "y": 252.2582773408825, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 184.9195556640625, + "height": 1.6022679018915937, + "seed": 357577356, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638348054411, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": 0.0344528515859526, + "gap": 10.57574462890625 + }, + "endBinding": { + "elementId": "dd52BjHfPMPRji9Tws7U-", + "focus": -0.039393828258510157, + "gap": 8.72607421875 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + 184.9195556640625, + -1.6022679018915937 + ] + ] + }, + { + "type": "arrow", + "version": 1173, + "versionNonce": 1248255756, + "isDeleted": false, + "id": "wRO0q9xKPHc8e8XPPsQWh", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 406.0439244722469, + "y": 246.80533728741074, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 157.86774649373126, + "height": 0.1417938392881979, + "seed": 656189364, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638348077725, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": 0.13736472619498497, + "gap": 16.306295254315614 + }, + "endBinding": { + "elementId": "Sa4OBd1ZjD6itohm7Ll8z", + "focus": -0.013200835330936087, + "gap": 14.437713623046875 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + -157.86774649373126, + 0.1417938392881979 + ] + ] + }, + { + "type": "text", + "version": 557, + "versionNonce": 995289780, + "isDeleted": false, + "id": "RbVSa4PnOgAMtzoKb-DhW", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 552.4987182617188, + "y": 212.27996826171875, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 188, + "height": 76, + "seed": 1989838604, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348059525, + "fontSize": 16, + "fontFamily": 3, + "text": "persist alerts state\n\n\nand recording rules", + "baseline": 72, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "text", + "version": 803, + "versionNonce": 1576507444, + "isDeleted": false, + "id": "ia2QzZNl_tuvfY3ymLjyJ", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 290.34130859375, + "y": 210.56927490234375, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 122, + "height": 19, + "seed": 157304972, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh" + ], + "updated": 1638347948032, + "fontSize": 16, + "fontFamily": 3, + "text": "execute rules", + "baseline": 15, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "arrow", + "version": 1471, + "versionNonce": 1361321140, + "isDeleted": false, + "id": "sxEhnxlbT7ldlSsmHDUHp", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 484.18669893674246, + "y": 302.3424013553929, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 1.0484739253853945, + "height": 84.72775855671654, + "seed": 1818348300, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638348083348, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": 0.010768924644894236, + "gap": 14.650750964767894 + }, + "endBinding": { + "elementId": "8-XFSbd6Zw96EUSJbJXZv", + "focus": -0.051051952959743775, + "gap": 11.437713623046818 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + 1.0484739253853945, + 84.72775855671654 + ] + ] + }, + { + "type": "text", + "version": 576, + "versionNonce": 2112088460, + "isDeleted": false, + "id": "E9Run6wCm2chQ6JHrmc_y", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 530.5612182617188, + "y": 318.60687255859375, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 122, + "height": 38, + "seed": 1836541708, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348023735, + "fontSize": 16, + "fontFamily": 3, + "text": "send alert \nnotifications", + "baseline": 34, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "text", + "version": 480, + "versionNonce": 1835119500, + "isDeleted": false, + "id": "ff5OkfgmkKLifS13_TFj3", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 291.37474060058594, + "y": 261.4861297607422, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 122, + "height": 19, + "seed": 264004620, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh" + ], + "updated": 1638347948032, + "fontSize": 16, + "fontFamily": 3, + "text": "restore state", + "baseline": 15, + "textAlign": "left", + "verticalAlign": "top" + } + ], + "appState": { + "gridSize": null, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} \ No newline at end of file diff --git a/app/vmalert/vmalert_cluster.png b/app/vmalert/vmalert_cluster.png new file mode 100644 index 000000000..184df339e Binary files /dev/null and b/app/vmalert/vmalert_cluster.png differ diff --git a/app/vmalert/vmalert_ha.excalidraw b/app/vmalert/vmalert_ha.excalidraw new file mode 100644 index 000000000..d77788219 --- /dev/null +++ b/app/vmalert/vmalert_ha.excalidraw @@ -0,0 +1,911 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "type": "rectangle", + "version": 906, + "versionNonce": 448716468, + "isDeleted": false, + "id": "VgBUzo0blGR-Ijd2mQEEf", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 302.2676696777344, + "y": 275.59356689453125, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 123.7601318359375, + "height": 72.13211059570312, + "seed": 1194011660, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "miEbzHxOPXe4PEYvXiJp5", + "rcmiQfIWtfbTTlwxqr1sl", + "P-dpWlSTtnsux-zr5oqgF", + "oAToSPttH7aWoD_AqXGFX", + "Bpy5by47XGKB4yS99ZkuA", + "wRO0q9xKPHc8e8XPPsQWh", + "sxEhnxlbT7ldlSsmHDUHp", + "m9_BptFOFxbV2sS_xJDu2", + "fsGFp4NW4JlrCdF0HR3uA", + "OTKoeHmKtqCxFArDbY-sP" + ], + "updated": 1638355221749 + }, + { + "type": "text", + "version": 762, + "versionNonce": 223660724, + "isDeleted": false, + "id": "e9TDm09y-GhPm84XWt0Jv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 318.0538024902344, + "y": 296.6778106689453, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 327273100, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "m9_BptFOFxbV2sS_xJDu2" + ], + "updated": 1638355102070, + "fontSize": 20, + "fontFamily": 3, + "text": "vmalert1", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 2067, + "versionNonce": 817347852, + "isDeleted": false, + "id": "dd52BjHfPMPRji9Tws7U-", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 329.6426086425781, + "y": 90.3275146484375, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 269.336669921875, + "height": 44.74725341796875, + "seed": 1779959692, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "Bpy5by47XGKB4yS99ZkuA", + "m9_BptFOFxbV2sS_xJDu2", + "S_dOHQrhGmu8SFJzobJK7" + ], + "updated": 1638355058507 + }, + { + "type": "text", + "version": 1717, + "versionNonce": 2040797452, + "isDeleted": false, + "id": "9TEzv0sVCHAkc46ou0oNF", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 349.01177978515625, + "y": 102.45901489257812, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 234, + "height": 24, + "seed": 1617178804, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "m9_BptFOFxbV2sS_xJDu2", + "S_dOHQrhGmu8SFJzobJK7" + ], + "updated": 1638355040938, + "fontSize": 20, + "fontFamily": 3, + "text": "victoriametrics:8428", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 1553, + "versionNonce": 39381044, + "isDeleted": false, + "id": "8-XFSbd6Zw96EUSJbJXZv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 173.67257690429688, + "y": 470.5100402832031, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 240.10644531249997, + "height": 44.74725341796875, + "seed": 99322124, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp", + "OTKoeHmKtqCxFArDbY-sP", + "XhPgLRBk-YhWAFcSQi9TJ", + "D6fkQH1E_MFbCuL697ArO" + ], + "updated": 1638355221749 + }, + { + "type": "text", + "version": 1309, + "versionNonce": 1112872844, + "isDeleted": false, + "id": "GUs816aggGqUSdoEsSmea", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 189.667236328125, + "y": 482.59979248046875, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 211, + "height": 24, + "seed": 1194745268, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355040938, + "fontSize": 20, + "fontFamily": 3, + "text": "alertmanager1:9093", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "text", + "version": 835, + "versionNonce": 2036483596, + "isDeleted": false, + "id": "RbVSa4PnOgAMtzoKb-DhW", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 525.83837890625, + "y": 147.33470153808594, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 188, + "height": 95, + "seed": 1989838604, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355040939, + "fontSize": 16, + "fontFamily": 3, + "text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n", + "baseline": 91, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "text", + "version": 777, + "versionNonce": 990468492, + "isDeleted": false, + "id": "E9Run6wCm2chQ6JHrmc_y", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 617.0690307617188, + "y": 376.9822692871094, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 122, + "height": 38, + "seed": 1836541708, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638355227582, + "fontSize": 16, + "fontFamily": 3, + "text": "send alert \nnotifications", + "baseline": 34, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 1112, + "versionNonce": 999136652, + "isDeleted": false, + "id": "mIu-d0lmShCxzMLD5iA_p", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 492.29505920410156, + "y": 272.3052215576172, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 123.7601318359375, + "height": 72.13211059570312, + "seed": 756416780, + "groupIds": [ + "jbBot-UNdMoWy2jPXC1u5" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "miEbzHxOPXe4PEYvXiJp5", + "rcmiQfIWtfbTTlwxqr1sl", + "P-dpWlSTtnsux-zr5oqgF", + "oAToSPttH7aWoD_AqXGFX", + "Bpy5by47XGKB4yS99ZkuA", + "wRO0q9xKPHc8e8XPPsQWh", + "sxEhnxlbT7ldlSsmHDUHp", + "S_dOHQrhGmu8SFJzobJK7", + "XhPgLRBk-YhWAFcSQi9TJ", + "Ar-hcDLlzVSoTPs2MaywO" + ], + "updated": 1638355153683 + }, + { + "type": "text", + "version": 970, + "versionNonce": 430760500, + "isDeleted": false, + "id": "ZqIR6SaLNDQl8s0zZbdnE", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 508.08119201660156, + "y": 293.38946533203125, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 1477404084, + "groupIds": [ + "jbBot-UNdMoWy2jPXC1u5" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355105020, + "fontSize": 20, + "fontFamily": 3, + "text": "vmalertN", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "id": "qZFezRGU4Chwxgvb2451t", + "type": "line", + "x": 449.93653869628906, + "y": 306.30010986328125, + "width": 19.48321533203125, + "height": 0.3865966796875, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 1833426828, + "version": 69, + "versionNonce": 871948300, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355040939, + "points": [ + [ + 0, + 0 + ], + [ + 19.48321533203125, + 0.3865966796875 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": null + }, + { + "type": "rectangle", + "version": 1649, + "versionNonce": 93981708, + "isDeleted": false, + "id": "gNHiZJKo0ap69ALDobtZ-", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 518.5596466064453, + "y": 471.4539489746094, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 240.10644531249997, + "height": 44.74725341796875, + "seed": 454422412, + "groupIds": [ + "fEAIeQ0DxLnI_rPlKPZqW" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp", + "fsGFp4NW4JlrCdF0HR3uA", + "Ar-hcDLlzVSoTPs2MaywO", + "D6fkQH1E_MFbCuL697ArO" + ], + "updated": 1638355153683 + }, + { + "type": "text", + "version": 1412, + "versionNonce": 75038348, + "isDeleted": false, + "id": "O-zgjZBvt4RI1PrkBNQnb", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 534.3148040771484, + "y": 483.543701171875, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 211, + "height": 24, + "seed": 1026268980, + "groupIds": [ + "fEAIeQ0DxLnI_rPlKPZqW" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355040939, + "fontSize": 20, + "fontFamily": 3, + "text": "alertmanagerN:9093", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "id": "m9_BptFOFxbV2sS_xJDu2", + "type": "arrow", + "x": 378.57185562792466, + "y": 262.1596984863281, + "width": 79.96146539019026, + "height": 111.53411865234375, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 141224884, + "version": 521, + "versionNonce": 283254540, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355102070, + "points": [ + [ + 0, + 0 + ], + [ + 79.96146539019026, + -111.53411865234375 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": -0.23996018586441184, + "gap": 13.433868408203125 + }, + "endBinding": { + "elementId": "dd52BjHfPMPRji9Tws7U-", + "focus": -0.14207100087207922, + "gap": 15.550811767578125 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "S_dOHQrhGmu8SFJzobJK7", + "type": "arrow", + "x": 558.1990515577129, + "y": 263.4271240234375, + "width": 88.18940317574186, + "height": 114.15780639648438, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 1004789812, + "version": 314, + "versionNonce": 1259171724, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355105019, + "points": [ + [ + 0, + 0 + ], + [ + -88.18940317574186, + -114.15780639648438 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "mIu-d0lmShCxzMLD5iA_p", + "focus": 0.4315094049079832, + "gap": 8.878097534179688 + }, + "endBinding": { + "elementId": "dd52BjHfPMPRji9Tws7U-", + "focus": 0.14840834302306677, + "gap": 14.194549560546875 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "fsGFp4NW4JlrCdF0HR3uA", + "type": "arrow", + "x": 356.6613630516658, + "y": 354.95416259765625, + "width": 278.74351860741064, + "height": 106.90020751953125, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 712384692, + "version": 334, + "versionNonce": 266493324, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355102070, + "points": [ + [ + 0, + 0 + ], + [ + 278.74351860741064, + 106.90020751953125 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": 0.7721210277890177, + "gap": 7.228485107421875 + }, + "endBinding": { + "elementId": "gNHiZJKo0ap69ALDobtZ-", + "focus": 0.4493598001028791, + "gap": 9.599578857421875 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "OTKoeHmKtqCxFArDbY-sP", + "type": "arrow", + "x": 361.02563221226757, + "y": 356.2252197265625, + "width": 95.3594006000182, + "height": 105.52130126953125, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 1992252596, + "version": 466, + "versionNonce": 472553100, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355221749, + "points": [ + [ + 0, + 0 + ], + [ + -95.3594006000182, + 105.52130126953125 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": -0.39325294425055324, + "gap": 8.499542236328125 + }, + "endBinding": { + "elementId": "8-XFSbd6Zw96EUSJbJXZv", + "focus": -0.400636314282374, + "gap": 8.763519287109375 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "XhPgLRBk-YhWAFcSQi9TJ", + "type": "arrow", + "x": 561.7189961327852, + "y": 349.238037109375, + "width": 266.271510370216, + "height": 112.383544921875, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 588869260, + "version": 453, + "versionNonce": 584723212, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355150131, + "points": [ + [ + 0, + 0 + ], + [ + -266.271510370216, + 112.383544921875 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "mIu-d0lmShCxzMLD5iA_p", + "focus": -0.7084007026732048, + "gap": 4.8007049560546875 + }, + "endBinding": { + "elementId": "8-XFSbd6Zw96EUSJbJXZv", + "focus": -0.4180430111580123, + "gap": 8.888458251953125 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "Ar-hcDLlzVSoTPs2MaywO", + "type": "arrow", + "x": 557.5999880535809, + "y": 352.71795654296875, + "width": 112.87813113656136, + "height": 106.3255615234375, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 1508782476, + "version": 331, + "versionNonce": 1137561908, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355153683, + "points": [ + [ + 0, + 0 + ], + [ + 112.87813113656136, + 106.3255615234375 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "mIu-d0lmShCxzMLD5iA_p", + "focus": 0.4358123206211808, + "gap": 8.280624389648438 + }, + "endBinding": { + "elementId": "gNHiZJKo0ap69ALDobtZ-", + "focus": 0.4783744286829653, + "gap": 12.410430908203125 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "WqHnv9_g3SdkLZuy4WSHa", + "type": "text", + "x": 186.5891876220703, + "y": 523.145263671875, + "width": 272, + "height": 19, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "sharp", + "seed": 1310147468, + "version": 281, + "versionNonce": 949157044, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355040939, + "text": "Alertmanagers in cluster mode", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "baseline": 15 + }, + { + "id": "yA0kgvdF71wZbHJ7Cg2p8", + "type": "rectangle", + "x": 159.41685485839844, + "y": 440.7666015625, + "width": 625.9590148925781, + "height": 110.13494873046878, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "sharp", + "seed": 1465986996, + "version": 366, + "versionNonce": 196040844, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355040939 + }, + { + "id": "D6fkQH1E_MFbCuL697ArO", + "type": "arrow", + "x": 422.4853057861328, + "y": 494.8779132311229, + "width": 90.5517578125, + "height": 0.12427004064892344, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 2025208204, + "version": 316, + "versionNonce": 2144102156, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638355040939, + "points": [ + [ + 0, + 0 + ], + [ + 90.5517578125, + 0.12427004064892344 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "8-XFSbd6Zw96EUSJbJXZv", + "focus": 0.08064204025505255, + "gap": 8.706283569335938 + }, + "endBinding": { + "elementId": "gNHiZJKo0ap69ALDobtZ-", + "focus": -0.05976220061952895, + "gap": 5.5225830078125 + }, + "startArrowhead": "arrow", + "endArrowhead": "arrow" + }, + { + "type": "text", + "version": 463, + "versionNonce": 2118914484, + "isDeleted": false, + "id": "p4EDvSKPqZzfM_SReybeq", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 274.6086883544922, + "y": 56.77886962890625, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 394, + "height": 19, + "seed": 1214462348, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355074872, + "fontSize": 16, + "fontFamily": 3, + "text": "VictoriaMetrics with deduplication enabled", + "baseline": 15, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 712, + "versionNonce": 737833612, + "isDeleted": false, + "id": "YfNFeOucMo8BJk2P2JRex", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 229.77923583984375, + "y": 227.84378051757812, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 448.5542297363281, + "height": 134.06329345703122, + "seed": 932207756, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355133244 + }, + { + "type": "text", + "version": 427, + "versionNonce": 1458527796, + "isDeleted": false, + "id": "oldM7Q_aJtpWd2jXQ0iKf", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 238.9442901611328, + "y": 230.862548828125, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 225, + "height": 38, + "seed": 2131899572, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638355259358, + "fontSize": 16, + "fontFamily": 3, + "text": "vmalerts with identical \nconfigurations", + "baseline": 34, + "textAlign": "left", + "verticalAlign": "top" + } + ], + "appState": { + "gridSize": null, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} \ No newline at end of file diff --git a/app/vmalert/vmalert_ha.png b/app/vmalert/vmalert_ha.png new file mode 100644 index 000000000..76ac521ab Binary files /dev/null and b/app/vmalert/vmalert_ha.png differ diff --git a/app/vmalert/vmalert_multicluster.excalidraw b/app/vmalert/vmalert_multicluster.excalidraw new file mode 100644 index 000000000..c020f262e --- /dev/null +++ b/app/vmalert/vmalert_multicluster.excalidraw @@ -0,0 +1,915 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "type": "rectangle", + "version": 791, + "versionNonce": 48874036, + "isDeleted": false, + "id": "VgBUzo0blGR-Ijd2mQEEf", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 289.6802978515625, + "y": 399.3895568847656, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 123.7601318359375, + "height": 72.13211059570312, + "seed": 1194011660, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "miEbzHxOPXe4PEYvXiJp5", + "rcmiQfIWtfbTTlwxqr1sl", + "P-dpWlSTtnsux-zr5oqgF", + "oAToSPttH7aWoD_AqXGFX", + "wRO0q9xKPHc8e8XPPsQWh", + "sxEhnxlbT7ldlSsmHDUHp", + "pD9DcILMxa6GaR1U5YyMO", + "HPEwr85wL4IedW0AgdArp", + "EyecK0YM9Cc8T6ju-nTOc" + ], + "updated": 1638347812431 + }, + { + "type": "text", + "version": 658, + "versionNonce": 1653816076, + "isDeleted": false, + "id": "e9TDm09y-GhPm84XWt0Jv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 311.22686767578125, + "y": 420.4738006591797, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 82, + "height": 24, + "seed": 327273100, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347796775, + "fontSize": 20, + "fontFamily": 3, + "text": "vmalert", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "middle" + }, + { + "type": "rectangle", + "version": 802, + "versionNonce": 995326644, + "isDeleted": false, + "id": "Sa4OBd1ZjD6itohm7Ll8z", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 603.05322265625, + "y": 228.65371704101562, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 126267060, + "groupIds": [ + "ek-pq3umtz1yN-J_-preq" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "he-SpFjCxEQEWpWny2kKP", + "-pjrKo16rOsasM8viZPJ-", + "MGdu6GDIPNBAaEUr0Gt-a" + ], + "updated": 1638347737174 + }, + { + "type": "text", + "version": 624, + "versionNonce": 707755700, + "isDeleted": false, + "id": "we766A079lfGYu2_aC4Pl", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 640.7635803222656, + "y": 241.02886962890625, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 478660236, + "groupIds": [ + "ek-pq3umtz1yN-J_-preq" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347701075, + "fontSize": 20, + "fontFamily": 3, + "text": "vmselect", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "text", + "version": 802, + "versionNonce": 1974403340, + "isDeleted": false, + "id": "ia2QzZNl_tuvfY3ymLjyJ", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 401.241943359375, + "y": 342.4627990722656, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 178, + "height": 38, + "seed": 157304972, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh" + ], + "updated": 1638347863144, + "fontSize": 16, + "fontFamily": 3, + "text": "execute aggregating\nrecording rules", + "baseline": 34, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "id": "jrFZeFNsxlss7IsEZpA-J", + "type": "rectangle", + "x": 594.1509857177734, + "y": 192.09194946289062, + "width": 397.1286010742188, + "height": 209.62742614746097, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "sharp", + "seed": 48379060, + "version": 665, + "versionNonce": 617456652, + "isDeleted": false, + "boundElementIds": [ + "pD9DcILMxa6GaR1U5YyMO" + ], + "updated": 1638347763716 + }, + { + "id": "6ibhLp94HJFIdfP5HCEv8", + "type": "text", + "x": 609.7205657958984, + "y": 199.81723022460938, + "width": 225, + "height": 19, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "sharp", + "seed": 1053164852, + "version": 256, + "versionNonce": 538595124, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638347761092, + "text": "VM cluster with raw data", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "baseline": 15 + }, + { + "type": "rectangle", + "version": 914, + "versionNonce": 1576666164, + "isDeleted": false, + "id": "R5v-yZCVJ97BkJqr0Qb7H", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 807.5664215087891, + "y": 287.8628387451172, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 1794212620, + "groupIds": [ + "BJNOAY1MY3Evr9B3qQtHf" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "he-SpFjCxEQEWpWny2kKP", + "-pjrKo16rOsasM8viZPJ-", + "bZXA8PH9gYu-clotqJ4f7", + "MGdu6GDIPNBAaEUr0Gt-a" + ], + "updated": 1638347737174 + }, + { + "type": "text", + "version": 725, + "versionNonce": 267455500, + "isDeleted": false, + "id": "pWRC_smX7TuOI8_8UrA4H", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 840.0209197998047, + "y": 300.2379913330078, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 105, + "height": 24, + "seed": 421856180, + "groupIds": [ + "BJNOAY1MY3Evr9B3qQtHf" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347701076, + "fontSize": 20, + "fontFamily": 3, + "text": "vmstorage", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 843, + "versionNonce": 1244193972, + "isDeleted": false, + "id": "EqROOfYulSPsZm7ovxfQN", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 604.9091339111328, + "y": 345.2847442626953, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 2043521972, + "groupIds": [ + "ls6uq-W9bbVBM_UxAuyba" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "bZXA8PH9gYu-clotqJ4f7" + ], + "updated": 1638347718537 + }, + { + "type": "text", + "version": 676, + "versionNonce": 143370892, + "isDeleted": false, + "id": "ddQH1nnmT7HbKW7Xmv4zx", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 642.6199798583984, + "y": 357.90354919433594, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 335223180, + "groupIds": [ + "ls6uq-W9bbVBM_UxAuyba" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "bZXA8PH9gYu-clotqJ4f7" + ], + "updated": 1638347701076, + "fontSize": 20, + "fontFamily": 3, + "text": "vminsert", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "id": "bZXA8PH9gYu-clotqJ4f7", + "type": "arrow", + "x": 785.2667708463345, + "y": 367.2342882272049, + "width": 99.87819315552736, + "height": 24.681162491468, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 1153029388, + "version": 420, + "versionNonce": 1726025228, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638347704644, + "points": [ + [ + 0, + 0 + ], + [ + 99.87819315552736, + -24.681162491468 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "EqROOfYulSPsZm7ovxfQN", + "focus": 0.5247890891198189, + "gap": 8.36404562660789 + }, + "endBinding": { + "elementId": "R5v-yZCVJ97BkJqr0Qb7H", + "focus": -0.6931056940383433, + "gap": 9.943033572650961 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "type": "arrow", + "version": 528, + "versionNonce": 1908259468, + "isDeleted": false, + "id": "MGdu6GDIPNBAaEUr0Gt-a", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 781.4456641707259, + "y": 248.777857603323, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 104.0940537386266, + "height": 27.60697400233846, + "seed": 1695061516, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638347737174, + "startBinding": { + "elementId": "Sa4OBd1ZjD6itohm7Ll8z", + "focus": -0.5921495247360469, + "gap": 6.398850205882127 + }, + "endBinding": { + "elementId": "R5v-yZCVJ97BkJqr0Qb7H", + "focus": 0.7021471697457312, + "gap": 11.478007139455713 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + 104.0940537386266, + 27.60697400233846 + ] + ] + }, + { + "type": "rectangle", + "version": 883, + "versionNonce": 845352076, + "isDeleted": false, + "id": "4pW8hvBu3bo1eMtFvg_gS", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 601.6520690917969, + "y": 473.7677536010742, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 1678097076, + "groupIds": [ + "3kSpFrIN3kg4jwjDKpNWw" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "he-SpFjCxEQEWpWny2kKP", + "-pjrKo16rOsasM8viZPJ-", + "5W90eDBjtfZvkSuQTG0Iw" + ], + "updated": 1638347777173 + }, + { + "type": "text", + "version": 703, + "versionNonce": 519371956, + "isDeleted": false, + "id": "w_i2hO06oLa0bWbyAfFzU", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 639.3624267578125, + "y": 486.14290618896484, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 340753036, + "groupIds": [ + "3kSpFrIN3kg4jwjDKpNWw" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347776748, + "fontSize": 20, + "fontFamily": 3, + "text": "vmselect", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 745, + "versionNonce": 879581324, + "isDeleted": false, + "id": "U5U-67wL5fPwvzlxOAF5A", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 592.7498321533203, + "y": 437.2059860229492, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 397.1286010742188, + "height": 209.62742614746097, + "seed": 912540724, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "pD9DcILMxa6GaR1U5YyMO" + ], + "updated": 1638347776748 + }, + { + "type": "text", + "version": 345, + "versionNonce": 1628116148, + "isDeleted": false, + "id": "0IGVrvICMVeZp2RCaqTeP", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "dotted", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 608.3194122314453, + "y": 444.93126678466797, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 291, + "height": 19, + "seed": 68700428, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347784373, + "fontSize": 16, + "fontFamily": 3, + "text": "VM cluster with aggregated data", + "baseline": 15, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 996, + "versionNonce": 2112461580, + "isDeleted": false, + "id": "9QMf0HXPE1W4M9S-zMJVO", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 806.1652679443359, + "y": 532.9768753051758, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 523607476, + "groupIds": [ + "Eb2kWfz3ZWBu8cNul0h_c" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "he-SpFjCxEQEWpWny2kKP", + "-pjrKo16rOsasM8viZPJ-", + "9WBH34em4CdU2OwzqYIl5", + "5W90eDBjtfZvkSuQTG0Iw" + ], + "updated": 1638347777173 + }, + { + "type": "text", + "version": 804, + "versionNonce": 1660832052, + "isDeleted": false, + "id": "WOrOD5vn6EtMUQRJomt3-", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 838.6197662353516, + "y": 545.3520278930664, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 105, + "height": 24, + "seed": 1321420684, + "groupIds": [ + "Eb2kWfz3ZWBu8cNul0h_c" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638347776748, + "fontSize": 20, + "fontFamily": 3, + "text": "vmstorage", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 925, + "versionNonce": 2052807604, + "isDeleted": false, + "id": "4dtJZpXEUxSK3biwFG7vd", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 603.5079803466797, + "y": 590.3987808227539, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 171.99359130859375, + "height": 44.74725341796875, + "seed": 1738524468, + "groupIds": [ + "punXEDFtHkSpcd9seAkZj" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh", + "9WBH34em4CdU2OwzqYIl5", + "EyecK0YM9Cc8T6ju-nTOc" + ], + "updated": 1638347812431 + }, + { + "type": "text", + "version": 756, + "versionNonce": 2040967820, + "isDeleted": false, + "id": "rAbyooo-X08-86qjoK0WR", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 641.2188262939453, + "y": 603.0175857543945, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 94, + "height": 24, + "seed": 948598284, + "groupIds": [ + "punXEDFtHkSpcd9seAkZj" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "9WBH34em4CdU2OwzqYIl5" + ], + "updated": 1638347776748, + "fontSize": 20, + "fontFamily": 3, + "text": "vminsert", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "arrow", + "version": 660, + "versionNonce": 1560678196, + "isDeleted": false, + "id": "9WBH34em4CdU2OwzqYIl5", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 783.8656172818814, + "y": 612.3483247872634, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 99.87819315552736, + "height": 24.681162491468, + "seed": 1141424308, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638347777173, + "startBinding": { + "elementId": "4dtJZpXEUxSK3biwFG7vd", + "focus": 0.5247890891198189, + "gap": 8.364045626608004 + }, + "endBinding": { + "elementId": "9QMf0HXPE1W4M9S-zMJVO", + "focus": -0.6931056940383404, + "gap": 9.943033572650847 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + 99.87819315552736, + -24.681162491468 + ] + ] + }, + { + "type": "arrow", + "version": 768, + "versionNonce": 1653264948, + "isDeleted": false, + "id": "5W90eDBjtfZvkSuQTG0Iw", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 780.0445106062728, + "y": 493.8918941633816, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 104.0940537386266, + "height": 27.60697400233846, + "seed": 1852298380, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638347777173, + "startBinding": { + "elementId": "4pW8hvBu3bo1eMtFvg_gS", + "focus": -0.5921495247360469, + "gap": 6.398850205882127 + }, + "endBinding": { + "elementId": "9QMf0HXPE1W4M9S-zMJVO", + "focus": 0.7021471697457312, + "gap": 11.478007139455713 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + 104.0940537386266, + 27.60697400233846 + ] + ] + }, + { + "id": "HPEwr85wL4IedW0AgdArp", + "type": "arrow", + "x": 423.70701599121094, + "y": 428.34434509277344, + "width": 179.54901123046875, + "height": 182.9937744140625, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 389863732, + "version": 47, + "versionNonce": 1364399028, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638347805500, + "points": [ + [ + 0, + 0 + ], + [ + 179.54901123046875, + -182.9937744140625 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": 0.6700023593531782, + "gap": 10.266586303710938 + }, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "EyecK0YM9Cc8T6ju-nTOc", + "type": "arrow", + "x": 424.7585906982422, + "y": 441.12806701660156, + "width": 174.29449462890625, + "height": 170.56607055664062, + "angle": 0, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "strokeSharpness": "round", + "seed": 981082124, + "version": 92, + "versionNonce": 1261546252, + "isDeleted": false, + "boundElementIds": null, + "updated": 1638347812431, + "points": [ + [ + 0, + 0 + ], + [ + 174.29449462890625, + 170.56607055664062 + ] + ], + "lastCommittedPoint": null, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "focus": -0.6826568395144794, + "gap": 11.318161010742188 + }, + "endBinding": { + "elementId": "4dtJZpXEUxSK3biwFG7vd", + "focus": -0.8207814548026305, + "gap": 4.45489501953125 + }, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "type": "text", + "version": 775, + "versionNonce": 95530764, + "isDeleted": false, + "id": "o-yIJ_WZzVubhbVODvhcC", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 407.27012634277344, + "y": 489.33091735839844, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 141, + "height": 19, + "seed": 775116812, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "wRO0q9xKPHc8e8XPPsQWh" + ], + "updated": 1638347824876, + "fontSize": 16, + "fontFamily": 3, + "text": "persist results", + "baseline": 15, + "textAlign": "left", + "verticalAlign": "top" + } + ], + "appState": { + "gridSize": null, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} \ No newline at end of file diff --git a/app/vmalert/vmalert_multicluster.png b/app/vmalert/vmalert_multicluster.png new file mode 100644 index 000000000..8b51f2546 Binary files /dev/null and b/app/vmalert/vmalert_multicluster.png differ diff --git a/app/vmalert/vmalert_single.excalidraw b/app/vmalert/vmalert_single.excalidraw new file mode 100644 index 000000000..e3d8c3d11 --- /dev/null +++ b/app/vmalert/vmalert_single.excalidraw @@ -0,0 +1,354 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "type": "rectangle", + "version": 795, + "versionNonce": 1195460364, + "isDeleted": false, + "id": "VgBUzo0blGR-Ijd2mQEEf", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 422.3502197265625, + "y": 215.55953979492188, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 123.7601318359375, + "height": 72.13211059570312, + "seed": 1194011660, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "miEbzHxOPXe4PEYvXiJp5", + "rcmiQfIWtfbTTlwxqr1sl", + "P-dpWlSTtnsux-zr5oqgF", + "oAToSPttH7aWoD_AqXGFX", + "Bpy5by47XGKB4yS99ZkuA", + "wRO0q9xKPHc8e8XPPsQWh", + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348116633 + }, + { + "type": "text", + "version": 660, + "versionNonce": 1034125236, + "isDeleted": false, + "id": "e9TDm09y-GhPm84XWt0Jv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 443.89678955078125, + "y": 236.64378356933594, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 82, + "height": 24, + "seed": 327273100, + "groupIds": [ + "iBaXgbpyifSwPplm_GO5b" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348116633, + "fontSize": 20, + "fontFamily": 3, + "text": "vmalert", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "middle" + }, + { + "type": "rectangle", + "version": 1690, + "versionNonce": 236788620, + "isDeleted": false, + "id": "dd52BjHfPMPRji9Tws7U-", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 344.9837951660156, + "y": 64.64248657226562, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 269.336669921875, + "height": 44.74725341796875, + "seed": 1779959692, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "Bpy5by47XGKB4yS99ZkuA" + ], + "updated": 1638348176044 + }, + { + "type": "text", + "version": 1331, + "versionNonce": 945335476, + "isDeleted": false, + "id": "9TEzv0sVCHAkc46ou0oNF", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 364.35296630859375, + "y": 76.77398681640625, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 234, + "height": 24, + "seed": 1617178804, + "groupIds": [ + "2Lijjn3PwPQW_8KrcDmdu" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348176045, + "fontSize": 20, + "fontFamily": 3, + "text": "victoriametrics:8428", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "rectangle", + "version": 1099, + "versionNonce": 671872012, + "isDeleted": false, + "id": "8-XFSbd6Zw96EUSJbJXZv", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 360.6495666503906, + "y": 382.2536315917969, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 240.10644531249997, + "height": 44.74725341796875, + "seed": 99322124, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348116633 + }, + { + "type": "text", + "version": 865, + "versionNonce": 635839156, + "isDeleted": false, + "id": "GUs816aggGqUSdoEsSmea", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 382.64422607421875, + "y": 394.3433837890625, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 199, + "height": 24, + "seed": 1194745268, + "groupIds": [ + "6obQBPHIfExBKfejeLLVO" + ], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348116633, + "fontSize": 20, + "fontFamily": 3, + "text": "alertmanager:9093", + "baseline": 19, + "textAlign": "center", + "verticalAlign": "top" + }, + { + "type": "arrow", + "version": 2444, + "versionNonce": 361351692, + "isDeleted": false, + "id": "Bpy5by47XGKB4yS99ZkuA", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 486.0465703465111, + "y": 204.98379516601562, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 0.7962088410123442, + "height": 86.86798095703125, + "seed": 357577356, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638348176045, + "startBinding": { + "elementId": "VgBUzo0blGR-Ijd2mQEEf", + "gap": 10.57574462890625, + "focus": 0.0344528515859526 + }, + "endBinding": { + "elementId": "dd52BjHfPMPRji9Tws7U-", + "gap": 8.72607421875, + "focus": -0.039393828258510157 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + -0.7962088410123442, + -86.86798095703125 + ] + ] + }, + { + "type": "text", + "version": 640, + "versionNonce": 1892951180, + "isDeleted": false, + "id": "RbVSa4PnOgAMtzoKb-DhW", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 500.609619140625, + "y": 134.11544799804688, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 188, + "height": 95, + "seed": 1989838604, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [], + "updated": 1638348163424, + "fontSize": 16, + "fontFamily": 3, + "text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n", + "baseline": 91, + "textAlign": "left", + "verticalAlign": "top" + }, + { + "type": "arrow", + "version": 1472, + "versionNonce": 768565516, + "isDeleted": false, + "id": "sxEhnxlbT7ldlSsmHDUHp", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 486.5245361328125, + "y": 300.5478957313172, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 0.3521007656264601, + "height": 70.26802223743277, + "seed": 1818348300, + "groupIds": [], + "strokeSharpness": "round", + "boundElementIds": [], + "updated": 1638348116633, + "startBinding": { + "elementId": "E9Run6wCm2chQ6JHrmc_y", + "focus": 1.1925203824459496, + "gap": 11.77154541015625 + }, + "endBinding": { + "elementId": "8-XFSbd6Zw96EUSJbJXZv", + "focus": 0.0441077573536454, + "gap": 11.437713623046875 + }, + "lastCommittedPoint": null, + "startArrowhead": null, + "endArrowhead": "arrow", + "points": [ + [ + 0, + 0 + ], + [ + -0.3521007656264601, + 70.26802223743277 + ] + ] + }, + { + "type": "text", + "version": 577, + "versionNonce": 1646003636, + "isDeleted": false, + "id": "E9Run6wCm2chQ6JHrmc_y", + "fillStyle": "hachure", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "angle": 0, + "x": 498.29608154296875, + "y": 298.6573791503906, + "strokeColor": "#000000", + "backgroundColor": "transparent", + "width": 122, + "height": 38, + "seed": 1836541708, + "groupIds": [], + "strokeSharpness": "sharp", + "boundElementIds": [ + "sxEhnxlbT7ldlSsmHDUHp" + ], + "updated": 1638348116633, + "fontSize": 16, + "fontFamily": 3, + "text": "send alert \nnotifications", + "baseline": 34, + "textAlign": "left", + "verticalAlign": "top" + } + ], + "appState": { + "gridSize": null, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} \ No newline at end of file diff --git a/app/vmalert/vmalert_single.png b/app/vmalert/vmalert_single.png new file mode 100644 index 000000000..46f087449 Binary files /dev/null and b/app/vmalert/vmalert_single.png differ diff --git a/docs/vmalert.md b/docs/vmalert.md index 09e64097b..c6e04463c 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -6,7 +6,11 @@ sort: 4 `vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) -rules against configured address. It is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/) +rules against configured `-datasource.url`. For sending alerting notifications +vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag. +Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) +protocol and require `-remoteWrite.url` to be configured. +Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/) implementation and aims to be compatible with its syntax. ## Features @@ -22,12 +26,12 @@ implementation and aims to be compatible with its syntax. * Lightweight without extra dependencies. ## Limitations -* `vmalert` execute queries against remote datasource which has reliability risks because of network. -It is recommended to configure alerts thresholds and rules expressions with understanding that network request -may fail; +* `vmalert` execute queries against remote datasource which has reliability risks because of the network. +It is recommended to configure alerts thresholds and rules expressions with the understanding that network +requests may fail; * by default, rules execution is sequential within one group, but persistence of execution results to remote storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous -recording rule is reused in next one; +recording rule is reused in the next one; ## QuickStart @@ -37,13 +41,13 @@ git clone https://github.com/VictoriaMetrics/VictoriaMetrics cd VictoriaMetrics make vmalert ``` -The build binary will be placed to `VictoriaMetrics/bin` folder. +The build binary will be placed in `VictoriaMetrics/bin` folder. To start using `vmalert` you will need the following things: * list of rules - PromQL/MetricsQL expressions to execute; * datasource address - reachable MetricsQL endpoint to run queries against; * notifier address [optional] - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing, -aggregating alerts and sending notifications. +aggregating alerts, and sending notifications. * remote write address [optional] - [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) compatible storage to persist rules and alerts state info; * remote read address [optional] - MetricsQL compatible datasource to restore alerts state from. @@ -62,7 +66,7 @@ Then configure `vmalert` accordingly: Note there's a separate `remoteRead.url` to allow writing results of alerting/recording rules into a different storage than the initial data that's -queried. This allows using `vmalert` to aggregate data from a short-term, +queried. This allows using `vmalert` to aggregate data from a short-term, high-frequency, high-cardinality storage into a long-term storage with decreased cardinality and a bigger interval between samples. @@ -124,14 +128,14 @@ expression and then act according to the Rule type. There are two types of Rules: * [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - -Alerting rules allow to define alert conditions via `expr` field and to send notifications to +Alerting rules allow defining alert conditions via `expr` field and to send notifications to [Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty. * [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) - -Recording rules allow to define `expr` which result will be then backfilled to configured +Recording rules allow defining `expr` which result will be then backfilled to configured `-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. -`vmalert` forbids defining duplicates - rules with the same combination of name, expression and labels +`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels within one group. #### Alerting rules @@ -147,7 +151,7 @@ alert: expr: # Alerts are considered firing once they have been returned for this long. -# Alerts which have not yet fired for long enough are considered pending. +# Alerts which have not yet been fired for long enough are considered pending. # If param is omitted or set to 0 then alerts will be immediately considered # as firing once they return. [ for: | default = 0s ] @@ -192,19 +196,19 @@ For recording rules to work `-remoteWrite.url` must be specified. the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. -These are regular time series and may be queried from VM just as any other time series. +These are regular time series and maybe queried from VM just as any other time series. The state is stored to the configured address on every rule evaluation. * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state from configured address by querying time series with name `ALERTS_FOR_STATE`. -Both flags are required for proper state restoring. Restore process may fail if time series are missing +Both flags are required for proper state restoration. Restore process may fail if time series are missing in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) or received state doesn't match current `vmalert` rules configuration. ### Multitenancy -The are the following approaches exist for alerting and recording rules across +There are the following approaches exist for alerting and recording rules across [multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy): * To run a separate `vmalert` instance per each tenant. @@ -244,6 +248,106 @@ The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags). +### Topology examples + +The following sections are showing how `vmalert` may be used and configured +for different scenarios. + +Please note, not all flags in examples are required: +* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if +you have recording rules or want to store [alerts state](#alerts-state-on-restarts) on `vmalert` restarts; +* `-notifier.url` is optional and is needed only if you have alerting rules. + +#### Single-node VictoriaMetrics + +vmalert single + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions + -remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results + -remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart + -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger +``` + +The simplest configuration where one single-node VM server is used for +rules execution, storing recording rules results and alerts state. + +#### Cluster VictoriaMetrics + +vmalert cluster + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions + -remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results + -remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart + -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger +``` + +In [cluster mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) +VictoriaMetrics has separate components for writing and reading path: +`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions +and `vminsert` is used to persist recording rules results and alerts state. +Cluster mode could have multiple `vminsert` and `vmselect` components. In case when you want +to spread the load on these components - add balancers before them and configure +`vmalert` with balancer's addresses. Please, see more about VM's cluster architecture +[here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview). + +#### vmalert high availability + +vmalert ha + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions + -remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results + -remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart + -notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger + -notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers +``` + +For HA user can run multiple identically configured `vmalert` instances. +It means all of them will execute the same rules, write state and results to +the same destinations, and send alert notifications to multiple configured +Alertmanagers. + +To avoid recording rules results and alerts state duplication in VictoriaMetrics server +don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). + +Alertmanager will automatically deduplicate alerts with identical labels, so ensure that +all `vmalert`s are having the same config. + +Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/) +for Alertmanagers for better reliability. + +This example uses single-node VM server for the sake of simplicity. +Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics) if needed. + +#### Downsampling and aggregation via vmalert + +vmalert multi cluster + +`vmalert` configuration flags: +``` +./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard + -datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions + -remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results +``` + +Example shows how to build a topology where `vmalert` will process data from one cluster +and write results into another. Such clusters may be called as "hot" (low retention, +high-speed disks, used for operative monitoring) and "cold" (long term retention, +slower/cheaper disks, low resolution data). With help of `vmalert`, user can setup +recording rules to process raw data from "hot" cluster (by applying additional transformations +or reducing resolution) and push results to "cold" cluster. + +Please note, [replay](#rules-backfilling) feature may be used for transforming historical data. + +Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used. ### Web @@ -263,7 +367,7 @@ vmalert sends requests to `<-datasource.url>/render?format=json` during evaluati if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render` implements [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) for `format=json`. When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget -to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on query type. +to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on the query type. ## Rules backfilling @@ -322,11 +426,11 @@ to prevent cache pollution and unwanted time range boundaries adjustment during #### Recording rules -Result of recording rules `replay` should match with results of normal rules evaluation. +The result of recording rules `replay` should match with results of normal rules evaluation. #### Alerting rules -Result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts). +The result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts). To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression: ``` ALERTS{alertname="your_alertname", alertstate="firing"} @@ -339,7 +443,7 @@ There are following non-required `replay` flags: * `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request. In two words, it affects the max time range for every `/query_range` request. The higher the value, -the less requests will be issued during `replay`. +the fewer requests will be issued during `replay`. * `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries per rule before giving up. * `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining @@ -361,7 +465,7 @@ See full description for these flags in `./vmalert --help`. We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported metrics may be analyzed later. -Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. +Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. If you have suggestions for improvements or have found a bug - please open an issue on github or add a review to the dashboard. diff --git a/docs/vmalert_cluster.png b/docs/vmalert_cluster.png new file mode 100644 index 000000000..184df339e Binary files /dev/null and b/docs/vmalert_cluster.png differ diff --git a/docs/vmalert_ha.png b/docs/vmalert_ha.png new file mode 100644 index 000000000..76ac521ab Binary files /dev/null and b/docs/vmalert_ha.png differ diff --git a/docs/vmalert_multicluster.png b/docs/vmalert_multicluster.png new file mode 100644 index 000000000..8b51f2546 Binary files /dev/null and b/docs/vmalert_multicluster.png differ diff --git a/docs/vmalert_single.png b/docs/vmalert_single.png new file mode 100644 index 000000000..46f087449 Binary files /dev/null and b/docs/vmalert_single.png differ