mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-21 07:56:26 +01:00

vmalert: speed up reloading rules from object storage by verifying ob… (#755 )

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6210
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 974b7783ee)

2024-05-21 16:37:27 +02:00

132 KiB

Raw Blame History

sort

weight

title

aliases

100

CHANGELOG

docs

parent	weight
victoriametrics	100

/CHANGELOG.html

CHANGELOG

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation runs under the constant load generated by prometheus-benchmark and is used for testing latest releases.

tip

Update note 1: the -remoteWrite.multitenantURL command-line flag at vmagent was removed starting from this release. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details. Update note 2: the -streamAggr.dropInputLabels command-line flag at vmagent was renamed to -remoteWrite.streamAggr.dropInputLabels. -streamAggr.dropInputLabels is now used for global streaming aggregation.

SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.3. See the list of issues addressed in Go1.22.3.
FEATURE: dashboards/single: support selecting of multiple instances on the dashboard. See this issue for details.
FEATURE: dashboards/single: properly display version in the Stats row for the custom builds of VictoriaMetrics.
FEATURE: dashboards/single: add Network Usage panel to Resource Usage row.
FEATURE: dashboards/operator, dashboards/backupmanager and dashboard/tenant-statistic: update dashboard to be compatible with Grafana 10+ version.
FEATURE: dashboards/cluster: add new panel Concurrent selects to vmstorage row. The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details.
FEATURE: vmagent: support aggregation and deduplication configs before replicating data to configured -remoteWrite.url destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See this issue.
FEATURE: vmagent: add service discovery support for Vultr. See these docs and this issue.
FEATURE: vmagent: allow configuring -remoteWrite.disableOnDiskQueue and -remoteWrite.dropSamplesOnOverload cmd-line flags per each -remoteWrite.url. See this pull request. Thanks to @rbizos for implementaion!
FEATURE: vmagent: add labels path and url to metrics vmagent_remotewrite_push_failures_total and vmagent_remotewrite_samples_dropped_total. Now number of failed pushes and dropped samples can be tracked per -remoteWrite.url.
FEATURE: stream aggregation: add rate_sum and rate_avg aggregation outputs.
FEATURE: vmalert: reduce CPU usage when evaluating high number of alerting and recording rules.
FEATURE: vmalert: speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See this issue.
BUGFIX: vmui: fix bug that prevents the first query trace from expanding on click event. See this issue. The issue was introduced in v1.100.0 release.
BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.
BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.
BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.
BUGFIX: vmagent: prevent potential panic during stream aggregation if more than one --remoteWrite.streamAggr.dedupInterval is configured. See this issue.
BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.
BUGFIX: stream aggregation: set correct suffix <output>_prometheus for aggregation outputs increase_prometheus and total_prometheus. Before, outputs total and total_prometheus or increase and increase_prometheus had the same suffix.
BUGFIX: stream aggregation: prevent from excessive resource usage when stream aggregation config file is empty.
BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.
BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it's set to minimum supported value 1s.
BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.
BUGFIX: dashboards: fix AnnotationQueryRunner error in Grafana when executing annotations query against Prometheus backend. See this issue for details.
DEPRECATION: vmagent: removed deprecated -remoteWrite.multitenantURL flag from vmagent. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

v1.101.0

Released at 2024-04-26

FEATURE: MetricsQL: support filtering by multiple numeric constants via q == (C1, ..., CN) and q != (C1, ..., CN) syntax. For example, status_code == (200, 201, 300) returns status_code metrics with any of 200, 201 or 300 values, while status_code != (400, 404, 500) returns status_code metrics with all the values except of 400, 404 and 500.
FEATURE: VictoriaMetrics cluster: add support for fault domain awareness to vmselect. It can be configured to return full responses if up to -globalReplicationFactor - 1 fault domains (aka vmstorage groups) are unavailable. See this feature request and these docs.
FEATURE: all VictoriaMetrics enterprise components: add support for automatic issuing of TLS certificates for HTTPS server at -httpListenAddr via Let's Encrypt service. See these docs and this feature request.
FEATURE: vmsingle: support disable or log implicit conversions for subquery with cmd-line flags -search.disableImplicitConversion and -search.logImplicitConversion. See this feature request.
FEATURE: vmagent: support data replication additionally to sharding among remote storage systems if -remoteWrite.shardByURLReplicas=N command-line flag is set additionally to -remoteWrite.shardByURL command-line flag, where N is desired replication factor. This allows setting up data replication among failure domains when the replication factor is smaller than the number of failure domains. See these docs and this feature request.
FEATURE: vmagent: reduce CPU usage when sharding among remote storage systems is enabled.
FEATURE: stream aggregation: reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data.
FEATURE: vmagent: support DNS SRV addresses in -remoteWrite.url command-line option and in scrape target urls. For example, -remoteWrite.url=http://srv+victoria-metrics/api/v1/write automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See these docs and this feature request.
FEATURE: stream aggregation: allow skipping first N aggregation intervals via cmd-line flag -streamAggr.ignoreFirstIntervals for Single-node VictoriaMetrics or -remoteWrite.streamAggr.ignoreFirstIntervals for vmagent. See more details here.
FEATURE: vmauth: support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside url_prefix urls in the form http://srv+addr/path, where the addr is the DNS SRV address, which is automatically resolved to hostnames with TCP ports. See these docs for details.
FEATURE: vmauth: support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See these docs.
FEATURE: vmauth: support regex matching when routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.
FEATURE: vmui: optimize auto-suggestion performance for metric names when the database contains big number of unique time series.
FEATURE: vmui: in the Select component, user-entered values are now preserved on blur if they match options in the list.
FEATURE: vmui: auto-suggestion triggers at any cursor position in the query input. See this issue.
FEATURE: vmui: update error messages on the Query page for enhanced clarity. See this pull request.
BUGFIX: downsampling: skip unnecessary index lookups if downsampling wasn't set for ENT versions of VictoriaMetrics. Before, users of VictoriaMetrics ENT could have experience elevated CPU usage even if no downsampling was configured. The issue was introduced in v1.100.0.
BUGFIX: downsampling: properly populate downsampling metadata for data parts created by VictoriaMetrics ENT versions lower than v1.100.0. The bug could trigger the downsampling actions for parts that were downsampled already. This bug doesn't have any negative effect apart from spending extra CPU resources on the repeated downsampling. The issue was introduced in v1.100.0.
BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.
BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.
BUGFIX: vmalert: fix links with anchors in vmalert's UI. Starting from v1.99.0 vmalert could ignore anchors pointing to specific rule groups if search param was present in URL.
BUGFIX: vmauth: don't treat concurrency limit hit as an error of the backend. Previously, hitting the concurrency limit would increment both vmauth_concurrent_requests_limit_reached_total and vmauth_user_request_backend_errors_total counters. Now, only concurrency limit counter is incremented. Updates this issue.

v1.100.1

Released at 2024-04-11

Update note 1: This release contains the issue which could lead to extra CPU usage of storage component in ENT distribution of VictoriaMetrics. The issue is caused by downsampling per distinct sets of time series feature introduced in 1.100.0 ENT version. Please, upgrade to v1.101.0 or rollback to v1.98.0 ENT version if you're affected.

Update note 2: When upgrading to this release from v1.99.0 or v1.100.0 it is recommended to reset caches stored on disk according to these docs.

FEATURE: vmbackupmanager: allow specifying custom backup interval via -backupInterval command-line flag. See this feature request.
BUGFIX: properly register new entries in IndexDB when many new time series are ingested into VictoriaMetrics in a short period of time. See this and this issues. The bug has been introduced in v1.99.0.
BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.100.0

Released at 2024-04-04

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

Update note 1: the -datasource.lookback command-line flag at vmalert is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay option. See this issue for more details.

SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.
FEATURE: downsampling: add ability to configure distinct downsampling per distinct sets of time series and/or tenants. See this feature request and these docs.
FEATURE: vmauth: allow discovering ip addresses for backend instances hidden behind a shared hostname, via discover_backend_ips: true option. This allows evenly spreading load among backend instances. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP request headers via src_headers option at url_map. See these docs.
FEATURE: vmauth: add ability to read auth tokens from arbitrary HTTP request headers via -httpAuthHeader command-line flag. Previously auth tokens were read only from Authorization HTTP request header. See these docs for details.
FEATURE: vmauth: add ability to authorize by opaque HTTP request header value via auth_token option in -auth.config.
FEATURE: stream aggregation: reduce memory usage by up to 5x when aggregating over big number of unique time series. The memory usage reduction is most visible when stream deduplication is enabled.
FEATURE: stream aggregation: allow using -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval command-line flags without the need to specify -streamAggr.config and -remoteWrite.streamAggr.config. See these docs.
FEATURE: stream aggregation: add -streamAggr.dropInputLabels command-line flag, which can be used for dropping the listed labels from input samples before applying stream de-duplication and aggregation. This is faster and easier to use alternative to input_relabel_configs. See these docs.
FEATURE: stream aggregation: add dedup_interval option, which allows configuring individual deduplication intervals per each stream aggregation config.
FEATURE: stream aggregation: use the same logic in stream deduplication as in the deduplication at VictoriaMetrics. See this issue.
FEATURE: stream aggregation: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs. Thanks to @edma2 for the pull request.
FEATURE: stream aggregation: add an ability to ignore input samples with old timestamps outside the current aggregation interval. See these docs for details.
FEATURE: stream aggregation: add keep_metric_names option, which can be set at stream aggregation config in order to keep the original metric names in the output aggregated samples instead of using the default output metric naming scheme.
FEATURE: stream aggregation: align the time of aggregated data flush to the specified aggregation interval. For example, if interval is set to 1m, then the aggregated data will be flushed at the end of every minute. The alginment can be disabled by setting no_align_flush_to_interval: true option at stream aggregation config. See these docs for details.
FEATURE: stream aggregation: add unique_samples output, which can be used for calculating the number of unique sample values over the given interval.
FEATURE: stream aggregation: add increase_prometheus and total_prometheus outputs, which can be used for increase and total aggregations when the first sample of every new time series must be ignored.
FEATURE: stream aggregation: expose vm_streamaggr_flush_timeouts_total and vm_streamaggr_dedup_flush_timeouts_total counters at /metrics page, which can be used for detecting flush timeouts for stream aggregation states. Expose also vm_streamaggr_flush_duration_seconds and vm_streamaggr_dedup_flush_duration_seconds histograms for monitoring the real flush durations of stream aggregation states.
FEATURE: vmui: improve trace display for better visual separation of branches. See this pull request.
FEATURE: vmagent: ability to limit the ingestion rate via -maxIngestionRate command-line flag. See this pull request.
FEATURE: vmagent: use the provided -remoteWrite.tlsServerName as Host header in requests to -remoteWrite.url. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and the pull request.
FEATURE: vmagent: add -remoteWrite.shardByURL.ignoreLabels command-line flag, which can be used for specifying the ignored list of labels when sharding by -remoteWrite.url is enabled. Thanks to @edma2 for the idea and the pull request.
FEATURE: vmagent: automatically reload updated root CA certificates from files without the need to restart vmagent. See this issue.
FEATURE: optimize /api/v1/labels and /api/v1/label/.../values when match[] filters contains metric name. For example, /api/v1/label/instance/values?match[]=up now works much faster than before. See this issue.
FEATURE: vmagent: accept Prometheus remote write protocol data at /api/v1/push endpoint. This simplifies writing data to vmagent from Grafana Mimir. Thanks to @edma2 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for native protocol. See this feature request. Thanks to @khushijain21 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for VictoriaMetrics destination specified via --vm-* cmd-line flags used in InfluxDB, Remote Read protocol, OpenTSDB, Prometheus and Promscale migration modes.
FEATURE: vmctl: split explore phase in vm-native mode by time intervals when --vm-native-step-interval is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See this issue.
FEATURE: vmgateway: add -logInvalidAuthTokens command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See this feature request.
FEATURE: graphite: add support for aggregateSeriesLists, diffSeriesLists, multiplySeriesLists and sumSeriesLists functions. Thanks to @rbizos for the pull request.
FEATURE: OpenTelemetry: add -opentelemetry.usePrometheusNaming command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See these docs and this issue.
BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. The probability of this bug has been increased significantly after v1.99.0 because of optimizations related to registering new time series. See this and this issue.
BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.
BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.
BUGFIX: do not drop match[] filter at /api/v1/series if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set, since missing match[] filter breaks /api/v1/series requests.
BUGFIX: vmagent: return proper responses for AWS Firehose requests according to these docs. See this pull request and this issue.
BUGFIX: vmctl: properly parse TLS key and CA files for InfluxDB and OpenTSDB migration modes.
BUGFIX: vmui: fix VictoriaLogs UI query handling to correctly apply _time filter across all queries. See this issue.
BUGFIX: vmui: fix issue where step parameter wasn't updated after moving the focus outside the step input field in Firefox.
BUGFIX: vmui: fix freezing when pasting a query with autocomplete enabled. See this issue.
BUGFIX: vmui: fix auto-suggestion trigger issue after whitespace char. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with -search.maxLabelsAPIDuration duration. Before, -search.maxExportDuration value was used by mistake. The bug has been introduced in v1.99.0. Thanks to @kbweave for the pull request.
BUGFIX: properly wait for force merge to be completed during the shutdown. See this issue for the details.
BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.
BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.
BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.

v1.99.0

Released at 2024-03-01

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

FEATURE: MetricsQL: propagate label filters via all the label manipulation functions. For example, label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"} is now transformed to label_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}. This should reduce the amounts of time series processed during query execution.
FEATURE: MetricsQL: add count_values_over_time function. See this feature request.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_last_partition_parts metrics, which show the number of parts in the latest partition. These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_snapshots metric, which shows the current number of snapshots created via snapshot API.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: add -search.ignoreExtraFiltersAtLabelsAPI command-line flag, which can be used for reducing load on VictoriaMetrics when /api/v1/labels, /api/v1/label/.../values or /api/v1/series are queried with too broad extra_filters or extra_label, which match many time series. See these docs for details.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: allow limiting CPU and RAM usage at /api/v1/labels, /api/v1/label/.../values and /api/v1/series on systems with high churn rate via -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries command-line flags. See these docs for details.
FEATURE: vmagent: support reading Amazon CloudWatch metrics in OpenTelemetry format from Amazon Data Firehose.
FEATURE: vmagent: add support for client_id option into kuma_sd_configs in the same way as Prometheus does. See this pull request.
FEATURE: vmagent: add support for enable_compression option in scrape_configs in order to be compatible with Prometheus scrape configs. See this pull request and this feature request. Note that vmagent was always supporting disable_compression option before Prometheus added enable_compression option.
FEATURE: vmctl: support client-side TLS configuration for InfluxDB, Remote Read protocol and OpenTSDB. See this feature request. Thanks to @khushijain21 for pull requests 1, 2, 3.
FEATURE: vmui: preserve WITH templates when clicking the prettify query button at the right side of query input field. See this feature request.
FEATURE: vmui: allow filling gaps on graphs with interpolated lines as Grafana does. See this feature request and this pull request.
FEATURE: vmalert: support filtering by group, rule or labels in vmalert's UI for /groups and /alerts pages. See the pull request by @victoramsantos.
FEATURE: docker-compose: create a separate docker-compose environment for VictoriaLogs installation, including fluentbit and VictoriaLogs Grafana datasource. See these docs for details.
FEATURE: vmbackupmanager: wait for up 30 seconds before making a snapshot for backup if vmstorage is temporarily unavailalbe. This should prevent from vmbackupmanager termination in this case. See this feature request.
BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.98.0 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
BUGFIX: vmui: fix Popper display issues. See this pull request.
BUGFIX: vmselect: format time with milliseconds precision when __timestamp__:rfc3339 is specified for /api/v1/export/csv API response. See this issue for details.
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.98.0

Released at 2024-02-14

SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at -httpListenAddr (aka mTLS). See these docs and this feature request.
FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple -httpListenAddr command-line flags. For example, ./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428 starts VictoriaMetrics, which accepts incoming http requests at both some-host:12345 and localhost:8428. See this feature request.
FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, -remoteWrite.sendTimeout=',20s,' specifies three -remoteWrite.sendTimeout values - the first one and the last one have default values (30s in this case), while the second one is set to 20s.
FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.
FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka /api/beta/sketches endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request.
FEATURE: vmagent: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning the timeout needed for establishing TLS connections to -remoteWrite.url. Bigger tls handshake timeouts should reduce the probability of http: TLS handshake error from ...: EOF errors at the remote storage side under high load. See this issue.
FEATURE: VictoriaMetrics cluster: add -disableReroutingOnUnavailable command-line flag to vminsert, which can be used for reducing resource usage spikes at vmstorage nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request.
FEATURE: add -search.resetRollupResultCacheOnStartup command-line flag for resetting query cache on startup. See this feature request.
FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example, label_set(q1, "a", "b") + q2{c="d"} is automatically transformed to label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"} now. This should improve performance for such queries. See this issue.
FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.
FEATURE: dashboards/vmagent: add Targets scraped/s stat panel showing the number of targets scraped by the vmagent per-second.
FEATURE: dashboards/all: add new panel CPU spent on GC. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed.
FEATURE: vmalert: support filtering for /api/v1/rules API. See the pull request by @victoramsantos.
FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via -snapshot.tls* cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request.
FEATURE: vmui: add a time picker to the "Logs Explorer" page. See this issue.
BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from /-/reload endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.
BUGFIX: vmui: fix handling invalid timezone. See this issue.
BUGFIX: vmui: fix the bug where the select does not open. See this issue.
BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.
BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.
BUGFIX: dashboards: update Storage full ETA panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request.
BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.

v1.97.4

Released at 2024-04-19

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.
BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. See this and this issue.
BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.
BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.
BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.
BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.
BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.
BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.
BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.
BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.97.3

Released at 2024-03-01

BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
BUGFIX: take into account -search.maxUniqueTimeseries limit at /api/v1/labels, /api/v1/label/.../values and /api/v1/series. This limit was ignored at these endpoints starting from v1.97.0 when trying to address this issue. It has been appeared that this can result in high CPU usage when heaviweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return limit exceeded error to heavyweight queries instead of burning CPU.
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.97.2 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.97.2

Released at 2024-02-14

SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.

v1.97.1

Released at 2024-02-01

FEATURE: vmagent: add support for username_file option at basic_auth section in scrape_configs. See this feature request. Thanks to @wasim-nihal for the initial implementation.
BUGFIX: vmagent: fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in v1.97.0.
BUGFIX: fix runtime error: slice bounds out of range panic, which can occur during query execution. See this issue. The bug has been introduced in v1.97.0.
BUGFIX: MetricsQL: properly handle avg_over_time({some_filter}[d]) keep_metric_names queries, where some_filter matches multiple time series with multiple names, while d is bigger or equal to 3h. See this issue.
BUGFIX: dashboards/single: fix typo in query for version annotation which falsely produced many version change events.

v1.97.0

Released at 2024-01-30

SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.
FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.
FEATURE: graphite: add support for negative index in groupByNode and aliasByNode functions. Thanks to @rbizos for the pull request.
FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.
FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.
FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each -remoteWrite.url via the command-line flag -remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.
FEATURE: vmagent: add ability to set attach_metadata.node=true option for all the kubernetes_sd_configs defined at -promscrape.config via -promscrape.kubernetes.attachNodeMetadataAll command-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation.
FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying flush_on_shutdown: true setting in streaming aggregation config. See more details here.
FEATURE: streaming aggregation: expand %{ENV_VAR} placeholders in config files with the corresponding environment variable values.
FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
- -datasource.oauth2.endpointParams for -datasource.url
- -notifier.oauth2.endpointParams for -notifier.url
- -remoteRead.oauth2.endpointParams for -remoteRead.url
- -remoteWrite.oauth2.endpointParams for -remoteWrite.url
FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via src_hosts option at url_map. See these docs.
FEATURE: vmauth: expose vmauth_user_request_backend_errors_total and vmauth_unauthorized_user_request_backend_errors_total metrics, which track the number of failed requests because of backend errors. See this feature request.
FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via metric_labels section. See this feature request.
FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at -httpListenAddr already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.
FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to -pushmetrics.url. See push metrics docs.
FEATURE: all VictoriaMetrics components: add -metrics.exposeMetadata command-line flag, which allows displaying TYPE and HELP metadata at /metrics page exposed at -httpListenAddr. This may be needed when the /metrics page is scraped by collector, which requires the TYPE and HELP metadata such as Google Cloud Managed Prometheus.
FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using file:///path/to/file or http://host/path syntax for the following command-line flags: -configAuthKey, -deleteAuthKey, -flagsAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -httpAuth.password, -metricsAuthKey, -pprofAuthKey, -reloadAuthKey, -search.resetCacheAuthKey, -snapshotAuthKey. For example, -httpAuth.password=file:///path/to/password. See these docs for details.
FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
FEATURE: dashboards/single: change dashboard title from VictoriaMetrics to VictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.
FEATURE: vmctl: rename cmd-line flag vm-native-disable-retries to vm-native-disable-per-metric-migration to better reflect its meaning.
FEATURE: vmctl: add -vm-native-src-insecure-skip-verify and -vm-native-dst-insecure-skip-verify command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.
FEATURE: Alerting rules for VictoriaMetrics: add job label to DiskRunsOutOfSpace alerting rule, so it is easier to understand to which installation the triggered instance belongs.
FEATURE: vmstorage: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by -maxLabelsPerTimeseries or -maxLabelValueLen command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.
FEATURE: vmui: add the ability to export and import query reports:
- add a Query Analyzer page that allows you to build graphs from JSON data containing the results of executing a query request.
- add an Export query button to the graph that saves the result of executing the query in JSON. See this pull request.
FEATURE: vmui: add -vmui.defaultTimezone flag to set a default timezone. See this issue and these docs.
FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.
FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.
FEATURE: add VictoriaMetrics datasource to docker compose environment. See this pull request.
BUGFIX: properly return the list of matching label names and label values from /api/v1/labels and /api/v1/label/.../values when the database contains more than -search.maxUniqueTimeseries unique time series on the selected time range. Previously VictoriaMetrics could return the number of matching timeseries exceeds ... error in this case. See this issue.
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
BUGFIX: VictoriaMetrics cluster: properly return full results when -search.skipSlowReplicas command-line flag is passed to vmselect and when vmstorage groups are in use. Previously partial results could be returned in this case.
BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.
BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.
BUGFIX: vmalert: automatically add exported_ prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue.
BUGFIX: vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0. Previously, ALERTS_FOR_STATE was generated only for alerts with for > 0. The change aligns with Prometheus behavior. See more details in this issue.
BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with context cancelled error.
BUGFIX: MetricsQL: consistently sort results for q1 or q2 query, so they do not change colors with each refresh in Grafana. See this issue.
BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
BUGFIX: vmagent: exit if there is config syntax error in scrape_config_files when -promscrape.config.strictParse=true. See this issue.
BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.
BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.
BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in kubernetes_sd_configs. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.
BUGFIX: vmagent: properly follow redirects to another hostnames when follow_redirects: true option is set in scrape_configs while stream parsing mode is disabled. See this issue and this pull request.
BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.
BUGFIX: vmui: fix the display of autocomplete results and cache the results. See this issue and this issue.
BUGFIX: vmui: send step param for instant queries. The change reverts this issue due to reasons explained in this comment.
BUGFIX: vmui: fix the Enter key press issue in the Query field when using autocomplete. See this issue.
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
BUGFIX: vmselect: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.
BUGFIX: vmselect: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example, sum(avg_over_time(up[24h])). This bug has been introduced in v1.95.0. See this and this issues.

v1.96.0

See changes here

v1.95.1

See changes here

v1.95.0

See changes here