lib/storage: remove cache directory if it contains reset_cache_on_startup file

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447

docs/CHANGELOG.md

@@ -16,6 +16,7 @@ sort: 15
 * FEATURE: support durations anywhere in [MetricsQL queries](https://docs.victoriametrics.com/MetricsQL.html). For example, `sum_over_time(m[1h]) / 1h` is a valid query, which is equivalent to `sum_over_time(m[1h]) / 3600`.
 * FEATURE: support durations without suffxies in [MetricsQL queries](https://docs.victoriametrics.com/MetricsQL.html). For example, `rate(m[3600])` is a valid query, which is equivalent to `rate(m[1h])`.
 * FEATURE: add `is_set` label to `flag` metrics. This allows determining explicitly set command-line flags with the query `flag{is_set="true"}`.
+* FEATURE: add ability to remove caches stored inside `<-storageDataPath>/cache` on startup if `reset_cache_on_startup` file is present there. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447).
 
 * BUGFIX: vmagent: remove `{ %space %}` typo in `/targets` output. The typo has been introduced in v1.62.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408).
 * BUGFIX: vmagent: fix CSS styles on `/targets` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422).

docs/README.md

@@ -1405,6 +1405,11 @@ These limits are approximate, so VictoriaMetrics can underflow/overflow the limi
 * VictoriaMetrics ignores `NaN` values during data ingestion.
 
 
+## Cache removal
+
+VictoriaMetrics uses various internal caches. These caches are stored to `<-storageDataPath>/cache` directory during graceful shutdown (e.g. when VictoriaMetrics is stopped by sending `SIGINT` signal). The caches are read on the next VictoriaMetrics startup. Sometimes it is needed to remove such caches on the next startup. This can be performed by placing `reset_cache_on_startup` file inside the `<-storageDataPath>/cache` directory before the restart of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447) for details.
+
+
 ## Data migration
 
 Use [vmctl](https://docs.victoriametrics.com/vmctl.html) for data migration. It supports the following data migration types:

docs/Single-server-VictoriaMetrics.md

@@ -1409,6 +1409,11 @@ These limits are approximate, so VictoriaMetrics can underflow/overflow the limi
 * VictoriaMetrics ignores `NaN` values during data ingestion.
 
 
+## Cache removal
+
+VictoriaMetrics uses various internal caches. These caches are stored to `<-storageDataPath>/cache` directory during graceful shutdown (e.g. when VictoriaMetrics is stopped by sending `SIGINT` signal). The caches are read on the next VictoriaMetrics startup. Sometimes it is needed to remove such caches on the next startup. This can be performed by placing `reset_cache_on_startup` file inside the `<-storageDataPath>/cache` directory before the restart of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447) for details.
+
+
 ## Data migration
 
 Use [vmctl](https://docs.victoriametrics.com/vmctl.html) for data migration. It supports the following data migration types:

lib/storage/storage.go

@@ -169,6 +169,18 @@ func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySer
 		return nil, fmt.Errorf("cannot create a directory for the storage at %q: %w", path, err)
 	}
 
+	// Check whether the cache directory must be removed
+	// It is removed if it contains reset_cache_on_startup file.
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447 for details.
+	if fs.IsPathExist(s.cachePath + "/reset_cache_on_startup") {
+		logger.Infof("removing cache directory at %q, since it contains `reset_cache_on_startup` file...", s.cachePath)
+		var wg sync.WaitGroup
+		wg.Add(1)
+		fs.MustRemoveAllWithDoneCallback(s.cachePath, wg.Done)
+		wg.Wait()
+		logger.Infof("cache directory at %q has been successfully removed", s.cachePath)
+	}
+
 	// Protect from concurrent opens.
 	flockF, err := fs.CreateFlockFile(path)
 	if err != nil {