From 2f0b2b942b470924f767a85c31ca475866bd2e4a Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Fri, 23 Feb 2024 22:00:30 +0200 Subject: [PATCH] docs/Troubleshooting.md: add `Too much disk space used` chapter` --- docs/Troubleshooting.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index a2f307d1d9..815238803f 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -20,6 +20,7 @@ This document contains troubleshooting guides for most common issues when workin - [Slow queries](#slow-queries) - [Out of memory errors](#out-of-memory-errors) - [Cluster instability](#cluster-instability) +- [Too much disk space used](#too-much-disk-space-used) - [Monitoring](#monitoring) ## General troubleshooting checklist @@ -441,6 +442,26 @@ and [cluster resizing and scalability docs](https://docs.victoriametrics.com/Clu for details. +## Too much disk space used + +If too much disk space is used by a [single-node VictoriaMetrics](https://docs.victoriametrics.com/) or by `vmstorage` component +at [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html), then please check the following: + +- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots) + and [snapshot troubleshooting](https://docs.victoriametrics.com/#snapshot-troubleshooting). + +- Under normal conditions the size of `<-storageDataPath>/indexdb` folder must be smaller than the size of `<-storageDataPath>/data` folder, where `-storageDataPath` + is the corresponding command-line flag value. This can be checked by the following query if [VictoriaMetrics monitoring](#monitoring) is properly set up: + ```metricsql + sum(vm_data_size_bytes{type=~"indexdb/.+"}) without(type) + / + sum(vm_data_size_bytes{type=~"(storage|indexdb)/.+"}) without(type) + ``` + If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issue, + which results in excess disk space usage for both `indexdb` and `data` folders under `-storageDataPath` folder. + The solution is to identify and fix the source of high churn rate with [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer). + + ## Monitoring Having proper [monitoring](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#monitoring)