From 5f95f9d4538a8b69296a52e82c81a155702d1efb Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Sat, 1 Apr 2023 08:50:27 +0200 Subject: [PATCH] lib/storage: check for free disk space before opening tables (#4035) * lib/storage: check for free disk space before opening tables We check for free disk space before call to `openTable`, so `Storage` can be set to ReadOnly before mergeWorkers start. Before the change, there was a chance that merges will start even if Storage has to start in ReadOnly mode because of `-storage.minFreeDiskSpaceBytes` limit. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023 Signed-off-by: hagen1778 * lib/storage: chore Signed-off-by: hagen1778 * Update lib/storage/storage.go --------- Signed-off-by: hagen1778 Co-authored-by: Aliaksandr Valialkin --- docs/CHANGELOG.md | 1 + lib/storage/storage.go | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 69517fced5..b58a718b36 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -46,6 +46,7 @@ created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or n * BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See [these docs](https://docs.victoriametrics.com/#environment-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3999). * BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in [v1.86.0](https://docs.victoriametrics.com/CHANGELOG.html#v1860). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966). * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). +* BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023). * BUGFIX: properly support comma-separated filters inside [retention filters](https://docs.victoriametrics.com/#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). * BUGFIX: verify response code when fetching configuration files via HTTP. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034). * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): replace empty labels with "" instead of "" during templating, as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012). diff --git a/lib/storage/storage.go b/lib/storage/storage.go index 6246513078..2835fa9164 100644 --- a/lib/storage/storage.go +++ b/lib/storage/storage.go @@ -262,6 +262,10 @@ func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySer s.setDeletedMetricIDs(dmisCurr) s.updateDeletedMetricIDs(dmisPrev) + // check for free disk space before opening the table + // to prevent unexpected part merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023 + s.startFreeDiskSpaceWatcher() + // Load data tablePath := filepath.Join(path, dataDirname) tb, err := openTable(tablePath, s) @@ -274,7 +278,6 @@ func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySer s.startCurrHourMetricIDsUpdater() s.startNextDayMetricIDsUpdater() s.startRetentionWatcher() - s.startFreeDiskSpaceWatcher() return s, nil }