From 8b907fffdbcc9dba4fa10fea0d37089cf1fba39c Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Fri, 10 Mar 2023 10:53:48 +0100 Subject: [PATCH] vmalert: support concurrent reading from object storage (#532) * vmalert: support concurrent reading from object storage Config reading from GCS or S3 can be slow if object storage contains a big number of files. Object storages are usually fast for downloading and are slow for individual operations. If there would be thousands of files to read, vmalert could spend significant time for retrieving those because it is done sequentially. The change introduces ability to read configs from object storage concurrently. By default, both GCS and S3 are now read with 50 concurrent readers. This significantly reduces the load time: * loading 500 files with concurrency=1 takes 27s * loading 500 files with concurrency=50 takes <1s * vmalert: add note to Changelog * vmalert: cleanup * vmalert: use ticker properly * app/vmalert: improve status reporting during config loading * vmalert: support concurrent reading from object storage Config reading from GCS or S3 can be slow if object storage contains a big number of files. Object storages are usually fast for downloading and are slow for individual operations. If there would be thousands of files to read, vmalert could spend significant time for retrieving those because it is done sequentially. The change introduces ability to read configs from object storage concurrently. By default, both GCS and S3 are now read with 50 concurrent readers. This significantly reduces the load time: * loading 500 files with concurrency=1 takes 27s * loading 500 files with concurrency=50 takes <1s * app/vmalert: make linter happy --- docs/CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index c160e5e5ce..cf33aaccdc 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -20,9 +20,10 @@ The following tip changes can be tested by building VictoriaMetrics components f * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): increase the default value for `--remote-read-http-timeout` command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3879). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): migrate series one-by-one in [vm-native mode](https://docs.victoriametrics.com/vmctl.html#native-protocol). This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3859) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add `--vm-native-src-headers` and `--vm-native-dst-headers` command-line flags, which can be used for setting custom HTTP headers during [vm-native migration mode](https://docs.victoriametrics.com/vmctl.html#native-protocol). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3906). -* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): log number of configration files found for each specified `-rule` command-line flag. * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add `--vm-native-disable-http-keep-alive` command-line flags to allow `vmctl` to use non-persistent HTTP connections in [vm-native migration mode](https://docs.victoriametrics.com/vmctl.html#native-protocol). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3909). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add `--vm-native-src-bearer-token` and `--vm-native-dst-bearer-token` command-line flags, which can be used for setting custom HTTP headers during [vm-native migration mode](https://docs.victoriametrics.com/vmctl.html#native-protocol). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3835). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): log number of configration files found for each specified `-rule` command-line flag. +* FEATURE: [vmalert enterprise](https://docs.victoriametrics.com/vmalert.html): concurrently [read config files from S3, GCS or S3-compatible object storage](https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage). This significantly improves config load speed for cases when there are thousands of files to read from the object storage. * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): fix panic when [writing data to Kafka](https://docs.victoriametrics.com/vmagent.html#writing-metrics-to-kafka). The panic has been introduced in [v1.88.0](https://docs.victoriametrics.com/CHANGELOG.html#v1880). * BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during [background merge](https://docs.victoriametrics.com/#storage). The issue has been introduced at [v1.85.0](https://docs.victoriametrics.com/CHANGELOG.html#v1850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897).