From 67d166f181430b5dd71ed5d6efff7ab01e46e584 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Mon, 5 Feb 2024 14:41:25 +0200 Subject: [PATCH] docs/Cluster-VictoriaMetrics.md: move the `Improve re-routing performance during restart` to more appropriate place Previoulsy it was mistakenly inserted between `No downtime strategy` and 'Minimum downtime strategy' chapters. This is a follow-up for 37997abd148e41baad5e855d025b3ec3eca91662 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5293 --- docs/Cluster-VictoriaMetrics.md | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md index f2d609eb75..244fc16bdd 100644 --- a/docs/Cluster-VictoriaMetrics.md +++ b/docs/Cluster-VictoriaMetrics.md @@ -486,20 +486,6 @@ This strategy allows upgrading the cluster without downtime if the following con If at least a single condition isn't met, then the rolling restart may result in cluster unavailability during the config update / version upgrade. In this case the following strategy is recommended. -#### Improving re-routing performance during restart - -`vmstorage` nodes may experience increased usage for CPU, RAM and disk IO during -[rolling restarts](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#no-downtime-strategy), -since they need to process higher load when some of `vmstorage` nodes are temporarily unavailable in the cluster. -It is possible to reduce resource usage spikes by running more `vminsert` nodes and by passing bigger values -to `-storage.vminsertConnsShutdownDuration` (available from [v1.95.0](https://docs.victoriametrics.com/CHANGELOG.html#v1950)) -command-line flag at `vmstorage` nodes. -In this case `vmstorage` increases the interval between gradual closing of `vminsert` connections during graceful shutdown. -This reduces data ingestion slowdown during rollout restarts. - -Make sure that the `-storage.vminsertConnsShutdownDuration` is smaller than the graceful shutdown timeout configured at the system which manages `vmstorage` -(e.g. Docker, Kubernetes, systemd, etc.). Otherwise the system may kill `vmstorage` node before it finishes gradual closing of `vminsert` connections. - ### Minimum downtime strategy 1. Gracefully stop all the `vminsert` and `vmselect` nodes in parallel. @@ -518,6 +504,23 @@ The `minimum downtime` strategy has the following benefits comparing to `no down - It allows minimizing the duration of config update / version upgrade for clusters with big number of nodes of for clusters with big `vmstorage` nodes, which may take long time for graceful restart. +## Improving re-routing performance during restart + +`vmstorage` nodes may experience increased usage for CPU, RAM and disk IO during +[rolling restarts](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#no-downtime-strategy), +since they need to process higher load when some of `vmstorage` nodes are temporarily unavailable in the cluster. +It is possible to reduce resource usage spikes by running more `vminsert` nodes and by passing bigger values +to `-storage.vminsertConnsShutdownDuration` (available from [v1.95.0](https://docs.victoriametrics.com/CHANGELOG.html#v1950)) +command-line flag at `vmstorage` nodes. +In this case `vmstorage` increases the interval between gradual closing of `vminsert` connections during graceful shutdown. +This reduces data ingestion slowdown during rollout restarts. + +Make sure that the `-storage.vminsertConnsShutdownDuration` is smaller than the graceful shutdown timeout configured at the system which manages `vmstorage` +(e.g. Docker, Kubernetes, systemd, etc.). Otherwise the system may kill `vmstorage` node before it finishes gradual closing of `vminsert` connections. + +See also [minimum downtime strategy](#minimum-downtime-strategy). + + ## Cluster availability VictoriaMetrics cluster architecture prioritizes availability over data consistency.