diff --git a/docs/guides/multi-regional-setup-dedicated-regions.md b/docs/guides/multi-regional-setup-dedicated-regions.md new file mode 100644 index 000000000..15b21c9c2 --- /dev/null +++ b/docs/guides/multi-regional-setup-dedicated-regions.md @@ -0,0 +1,83 @@ +# Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring + +### Scenario + +Let's cover the case. You have multiple regions with workloads and want to collect metrics. + +The monitoring setup is in the dedicated regions as shown below: + +

+ Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring +

+ +Every workload region (Earth, Mars, Venus) has a vmagent that sends data to multiple regions with a monitoring setup. +The monitoring setup (Ground Control 1,2) contains VictoriaMetrics Time Series Database(TSDB) cluster or single. + +Using this schema, you can achieve: + +* Global Querying View +* Querying all metrics from one monitoring installation +* High Availability +* You can lose one region, but your experience will be the same. +* Of course, that means you duplicate your traffic twice. + +### How to write the data to Ground Control regions + +* You need to specify two remote write URLs in vmagent configuration + +```bash +/vmagent-prod +-remoteWrite.url= +-remoteWrite.url= +``` + +* If you use the Pull model for data collection, please specify -promscrape.config parameter as well + +Here is a Quickstart guide for [vmagent](https://docs.victoriametrics.com/vmagent.html#quick-start) + +### How to read the data from Ground Control regions + +You can use one of the following options: + +1. Regional endpoints - use one regional endpoint as default and switch to another if there is an issue. +2. Load balancer - that sends queries to a particular region. The benefit and disadvantage of this setup is that it's simple. +3. Promxy - proxy that reads data from multiple Prometheus-like sources. It allows reading data more intelligently to cover the region's unavailability out of the box. It doesn't support MetricsQL yet (please check this issue). +4. Global vmselect in cluster setup - you can set up an additional subset of vmselects that knows about all storages in all regions. + * The deduplication in 1ms on the vmselect side must be turned on. This setup allows you to query data using MetricsQL. + * The downside is that vmselect waits for a response from all storages in all regions. + + +### High Availability + +The data is duplicated twice, and every region contains a full copy of the data. That means one region can be offline. + +You don't need to set up a replication factor using the VictoriaMetrics cluster. + +### Alerting + +You can set up vmalert in each Ground control region that evaluates recording and alerting rules. As every region contains a full copy of the data, you don't need to synchronize recording rules from one region to another. + +For alert deduplication, please use [cluster mode in Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/#high-availability). + +We also recommend adopting these alerts: + +* VictoriaMetrics Single - [https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) +* VictoriaMetrics Cluster - [https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/deployment/docker/alerts.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/deployment/docker/alerts.yml) + +### Monitoring + +An additional VictoriaMetrics single can be set up in every region, scraping metrics from the main TSDB. + +You also may evaluate the option to send these metrics to the neighbour region to achieve HA. + +Additional context +* VictoriaMetrics Single - [https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#monitoring](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#monitoring) +* VictoriaMetrics Cluster - [https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#monitoring](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#monitoring) + + +### What more can we do? + +Setup vmagents in Ground Control regions. That allows it to accept data close to storage and add more reliability if storage is temporarily offline. diff --git a/docs/guides/multi-regional-setup-dedicated-regions.png b/docs/guides/multi-regional-setup-dedicated-regions.png new file mode 100644 index 000000000..83b41c227 Binary files /dev/null and b/docs/guides/multi-regional-setup-dedicated-regions.png differ