mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 20:37:12 +01:00

app/vmctl: add flag where use can define path to the source remote read protocol (#4744 )

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655

2023-08-01 16:43:00 +02:00

CHANGELOG

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

tip

FEATURE: MetricsQL: add share_eq_over_time(m[d], eq) function for calculating the share (in the range [0...1]) of raw samples on the given lookbehind window d, which are equal to eq. See this feature request. Thanks to @Damon07 for the pull request.
FEATURE: vmalert: remove deprecated in v1.61.0 -rule.configCheckInterval command-line flag. Use -configCheckInterval command-line flag instead.
FEATURE: vmalert: remove support of deprecated web links of /api/v1/<groupID>/<alertID>/status form in favour of /api/v1/alerts?group_id=<>&alert_id=<> links. Links of /api/v1/<groupID>/<alertID>/status form were deprecated in v1.79.0. See this issue for details.
FEATURE: vmctl: allow disabling binary export API protocol via -vm-native-disable-binary-protocol cmd-line flag when migrating data from VictoriaMetrics. Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need to be configured at -vm-native-src-addr side and -vm-native-disable-binary-protocol should be set on vmctl side.
FEATURE: vmctl: add support of week step for time-based chunking migration. See this issue.
FEATURE: vmctl: do not add /api/v1/read suffix to remote read storage address defined by --remote-read-src-addr if a --remote-read-disable-path-append command-line flag is set. It allows an overriding path for remote-read API via --remote-read-src-addr. See this issue.
BUGFIX: vmagent: use local scrape timestamps for the scraped metrics unless honor_timestamps: true option is explicitly set at scrape_config. This fixes gaps for metrics collected from cadvisor or similar exporters, which export metrics with invalid timestamps. See this issue and this comment for details. The issue has been introduced in v1.68.0.
BUGFIX: vmbackupmanager: fix panic when creating a backup to a local filesystem on Windows. See this issue.

v1.92.1

Released at 2023-07-28

BUGFIX: vmalert: revert unit test feature for alerting and recording rules introduced in this pull request. See the following change.

v1.92.0

Released at 2023-07-27

Update note: starting from this release, stream aggregation writes the following samples to the configured remote storage by default:

aggregated samples;
the original input samples, which match zero match options from the provided config.

Previously only aggregated samples were written to the storage by default. The previous behavior can be restored in the following ways:

by passing -streamAggr.dropInput command-line flag to single-node VictoriaMetrics;
by passing -remoteWrite.streamAggr.dropInput command-line flag per each configured -remoteWrite.streamAggr.config at vmagent.

SECURITY: upgrade base docker image (alpine) from 3.18.0 to 3.18.2. See alpine 3.18.2 release notes.
SECURITY: upgrade Go builder from Go1.20.5 to Go1.20.6. See the list of issues addressed in Go1.20.6.
FEATURE: reduce memory usage by up to 5x for setups with high churn rate and long retention. See the description for this change and this issue for details.
FEATURE: reduce spikes in CPU and disk IO usage during indexdb rotation (aka inverted index), which is performed once per -retentionPeriod. The new algorithm gradually pre-populates newly created indexdb during the last hour before the rotation. The number of pre-populated series in the newly created indexdb can be monitored via vm_timeseries_precreated_total metric. This should resolve this issue.
FEATURE: MetricsQL: allow selecting time series matching at least one of multiple or filters. For example, {env="prod",job="a" or env="dev",job="b"} selects series with either {env="prod",job="a"} or {env="dev",job="b"} labels. This functionality allows passing the selected series to rollup functions without the need to use subqueries. See these docs.
FEATURE: MetricsQL: add ability to preserve metric names for binary operation results via keep_metric_names modifier. For example, ({__name__=~"foo|bar"} / 10) keep_metric_names leaves foo and bar metric names in division results. See these docs. This helps to address issues like this one.
FEATURE: MetricsQL: add ability to copy all the labels from one side of many-to-one operations by specifying * inside group_left() or group_right(). Also allow adding a prefix for copied label names via group_left(*) prefix "..." syntax. For example, the following query copies Kubernetes namespace labels to kube_pod_info series and adds ns_ prefix for the copied label names: kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels. The labels from on() list aren't prefixed. This feature resolves this and that questions at StackOverflow.
FEATURE: MetricsQL: add ability to specify durations via WITH templates. Examples:
- WITH (w = 5m) m[w] is automatically transformed to m[5m]
- WITH (f(window, step, off) = m[window:step] offset off) f(5m, 10s, 1h) is automatically transformed to m[5m:10s] offset 1h Thanks to @lujiajing1126 for the initial idea and implementation. See this feature request.
FEATURE: vmui: added a new page with the list of currently running queries. See this issue and these docs.
FEATURE: vmagent: add support for data ingestion via OpenTelemetry protocol. See these docs, this feature request and this pull request.
FEATURE: vmagent: allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable stream aggregation, when samples for the same time series must be aggregated by the same vmagent instance at the second level. See these docs and this feature request for details.
FEATURE: vmagent: allow configuring staleness interval in stream aggregation config. See this issue for details.
FEATURE: vmagent: allow specifying a list of series selectors inside if option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See these docs.
FEATURE: stream aggregation: allow specifying a list of series selectors inside match option of stream aggregation configs. The input sample is aggregated when at least a single series selector matches. See this feature request.
FEATURE: stream aggregation: preserve input samples, which match zero match options from the configured aggregations. Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing -streamAggr.dropInput command-line flag to single-node VictoriaMetrics or by passing -remoteWrite.streamAggr.dropInput command-line flag to vmagent.
FEATURE: vmctl: add verbose output for docker installations or when TTY isn't available. See this issue.
FEATURE: vmctl: interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See this pull request.
FEATURE: vmctl: update backoff policy on retries to reduce probability of overloading for source or destination databases. See this issue.
FEATURE: vmstorage: suppress "broken pipe" and "connection reset by peer" errors for search queries on vmstorage side. See this and this commits.
FEATURE: Official Grafana dashboards for VictoriaMetrics: add panel for tracking rate of syscalls while writing or reading from disk via process_io_(read|write)_syscalls_total metrics.
FEATURE: accept timestamps in milliseconds at start, end and time query args in Prometheus querying API. See these docs and this feature request.
FEATURE: vmalert: update retry policy for pushing data to -remoteWrite.url. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed -remoteWrite.retryMaxTime (default is 30s). When retry time is exceeded vmalert drops the data dedicated for -remoteWrite.url. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See -remoteWrite.retryMinInterval and -remoteWrite.retryMaxTime cmd-line flags.
FEATURE: vmalert: expose vmalert_remotewrite_send_duration_seconds_total counter, which can be used for determining high saturation of every connection to remote storage with an alerting query sum(rate(vmalert_remotewrite_send_duration_seconds_total[5m])) by(job, instance) > 0.9 * max(vmalert_remotewrite_concurrency) by(job, instance). This query triggers when a connection is saturated by more than 90%. This usually means that -remoteWrite.concurrency command-line flag must be increased in order to increase the number of concurrent writings into remote endpoint. See this feature request.
FEATUTE: vmalert: display the error message received during unsuccessful config reload in vmalert's UI. See this issue for details.
FEATUTE: vmalert: allow disabling of step param attached to instant queries. This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See this issue for details.
FEATUTE: vmalert: support option for "blackholing" alerting notifications if -notifier.blackhole cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See this issue for details. Thanks to @venkatbvc for the pull request.
FEATURE: vmalert: add unit test for alerting and recording rules, see more details here. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: allow overriding default GET params for rules with graphite datasource type, in the same way as it happens for prometheus type. See this issue.
FEATUTE: vmalert: support keep_firing_for field for alerting rules. See docs updated here and this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmauth: expose vmauth_user_request_duration_seconds and vmauth_unauthorized_user_request_duration_seconds summary metrics for measuring requests latency per user.
FEATURE: vmbackup: show backup progress percentage in log during backup uploading. See this issue.
FEATURE: vmrestore: show restoring progress percentage in log during backup downloading. See this issue.
FEATURE: add ability to fine-tune Graphite API limits via the following command-line flags: -search.maxGraphiteTagKeys for limiting the number of tag keys returned from Graphite API for tags -search.maxGraphiteTagValues for limiting the number of tag values returned from Graphite API for tag values -search.maxGraphiteSeries for limiting the number of series (aka paths) returned from Graphite API for series See this issue.
BUGFIX: properly return series from /api/v1/series if it finds more than the limit series (limit is an optional query arg passed to this API). Previously the limit exceeded error error was returned in this case. See this issue.
BUGFIX: vmui: fix application routing issues and problems with manual URL changes. See this pull request and this issue.
BUGFIX: add validation for invalid partial RFC3339 timestamp formats in query and export APIs.
BUGFIX: vmctl: interrupt explore procedure in influx mode if vmctl found no numeric fields.
BUGFIX: vmctl: fix panic in case --remote-read-filter-time-start flag is not set for remote-read mode. This flag is now required to use remote-read mode. See this issue.
BUGFIX: vmctl: fix formatting issue, which could add superflouos s characters at the end of samples/s output during data migration. For example, it could write samples/ssssss. See this issue.
BUGFIX: vmalert: use RFC3339 time format in query args instead of unix timestamp for all issued queries to Prometheus-like datasources.
BUGFIX: vmalert: correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group.
BUGFIX: vmalert: reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time.
BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch.
BUGFIX: vmagent: close HTTP connections to service discovery servers when they are no longer needed. This should prevent from possible connection exhasution in some cases. See this issue.
BUGFIX: vmagent: do not show relabel debug links at the /targets page when vmagent runs with -promscrape.dropOriginalLabels command-line flag, since it has no the original labels needed for relabel debug. See this issue.
BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via pushgateway protocol. This fixes Prometheus golang client compatibility. See this issue.
BUGFIX: MetricsQL: properly parse binary operations with reserved words on the right side such as foo + (on{bar="baz"}). Previously such queries could lead to panic. See this issue.
BUGFIX: Official Grafana dashboards for VictoriaMetrics: display cache usage for all components on panel Cache usage % by type for cluster dashboard. Before, only vmstorage caches were shown.

v1.91.3

Released at 2023-06-30

SECURITY: upgrade Go builder from Go1.20.4 to Go1.20.5. See the list of issues addressed in Go1.20.5.
BUGFIX: vmagent: fix possible panic at shutdown when stream aggregation is enabled. See this pull request for details.
BUGFIX: vmagent: fixed service name detection for consulagent service discovery in case of a difference in service name and service id. See this issue for details.
BUGFIX: vmalert: retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request.
BUGFIX: vmalert: properly interrupt retry attempts on vmalert shutdown. Before, vmalert could have waited for all retries to finish for shutdown.
BUGFIX: vmbackupmanager: fix an issue with vmbackupmanager not being able to restore data from a backup stored in GCS. See this issue for details.
BUGFIX: VictoriaMetrics cluster: properly return error from /api/v1/query and /api/v1/query_range at vmselect when the -search.maxSamplesPerQuery or -search.maxSamplesPerSeries limit is exceeded. Previously incomplete response could be returned without the error if vmselect runs with -replicationFactor greater than 1. See this pull request.
BUGFIX: storage: prevent from possible crashloop after the migration from versions below v1.90.0 to newer versions. See this issue for details.
BUGFIX: vmui: fix a memory leak issue associated with chart updates. See this pull request.
BUGFIX: vmbackupmanager: fix removing storage data dir before restoring from backup.
BUGFIX: vmselect: wait for all vmstorage nodes to respond when the -replicationFactor flag is set bigger than > 1. Before, vmselect could have skip waiting for the slowest replicas to respond. This could have resulted in issues illustrated here. Now, this optimization is disabled by default and could be re-enabled by passing -search.skipSlowReplicas cmd-line flag to vmselect. See more details here.

v1.91.2

Released at 2023-06-02

BUGFIX: vmalert: fix nil map assignment panic in runtime introduced in this change.

v1.91.1

Released at 2023-06-01

FEATURE:vmagent: Adds follow_redirects at service discovery level of scrape configuration. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See this issue. Thanks to @Haleygo for the pull request.
BUGFIX: vmalert: Properly form path to static assets in WEB UI if http.pathPrefix set. See this issue.
BUGFIX: vmalert: Properly set datasource query params. See this issue. Thanks to @gsakun for the pull request.
BUGFIX: vmalert: properly return empty slices instead of nil for /api/v1/rules for groups with present name but absent rules. See this issue.
BUGFIX: vmauth: Properly handle LOCAL command for proxy protocol. See this issue.
BUGFIX: vmbackupmanager: Fixes crash on startup. See this issue.
BUGFIX: vmui: fix bug with custom URL in global settings not respecting tenantID change. See this issue.

v1.91.0

Released at 2023-05-18

SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See the list of issues addressed in Go1.20.4.
SECURITY: serve /robots.txt content to disallow indexing of the exposed instances by search engines. See this issue for details.
FEATURE: update docker compose environment to V2 in respect to V1 deprecation notice from June 2023. See Migrate to Compose V2.
FEATURE: deprecate -bigMergeConcurrency command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for background merges can be controlled via -smallMergeConcurrency command-line flag, though it isn't recommended to change this flag in general case.
FEATURE: do not execute the incoming request if it has been canceled by the client before the execution start. See this pull request.
FEATURE: support time formats with timezones. For example, 2024-01-02+02:00 means January 2, 2024 at +02:00 time zone. See these docs.
FEATURE: expose process_* metrics at /metrics page of all the VictoriaMetrics components under Windows OS. See this pull request.
FEATURE: reduce the amounts of unimportant INFO logging during VictoriaMetrics startup / shutdown. This should improve visibility for potentially important logs.
FEATURE: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See alpine 3.18.0 release notes.
FEATURE: VictoriaMetrics cluster: do not pollute logs with cannot read hello: cannot read message with size 11: EOF messages at vmstorage during TCP health checks performed by Consul or other services. See this issue.
FEATURE: vmagent: support the ability to filter consul_sd_configs targets in more optimal way via new filter option. See this feature request.
FEATURE: vmagent: add support for consulagent_sd_configs. See this feature request.
FEATURE: vmagent: emit a warning if too small value is passed to -remoteWrite.maxDiskUsagePerURL command-line flag. See this issue.
FEATURE: vmalert: add support of recursive globs for -rule and -rule.templates command-line flags by using ** in the glob pattern. See this issue.
FEATURE: vmalert: add ability to specify custom per-group HTTP headers sent to the configured notifiers. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: detect alerting rules which don't match any series. See these docs and this feature request.
FEATURE: vmalert: support loading rules via HTTP URL. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: add buttons for filtering groups/rules with errors or with no-match warning in web UI for page /groups. See this issue.
FEATURE: vmalert: do not retry remote-write requests for responses with 4XX status codes. This aligns with Prometheus remote write specification. Thanks to @MichaHoffmann for the pull request.
FEATURE: vmauth: add ability to filter incoming requests by IP. See these docs and this feature request.
FEATURE: vmauth: add ability to proxy requests to the specified backends for unauthorized users. See this feature request.
FEATURE: vmauth: add ability to specify default route for unmatched requests. See this feature request.
FEATURE: vmauth: retry POST requests on the remaining backends if the currently selected backend isn't reachable. See this issue.
FEATURE: vmui: add ability to compare the data for the previous day with the data for the current day at Cardinality Explorer. See this feature request.
FEATURE: vmui: display histograms as heatmaps in Metrics explorer. See this feature request.
FEATURE: vmui: add WITH template playground. See this feature request.
FEATURE: vmui: add ability to debug relabeling. See this feature request.
FEATURE: vmui: add an ability to copy and execute queries listed at top queries page. Also make more human readable the query duration column. See this feature request and this pull request.
FEATURE: vmui: increase default font size for better readability.
FEATURE: vmui: cardinality explorer: return back a table with labels containing the highest number of unique label values. See issue.
FEATURE: vmui: add notification icon for queries that do not match any time series. A warning icon appears next to the query field when the executed query does not match any time series. See this feature request.
FEATURE: vmbackup: add -s3StorageClass command-line flag for setting the storage class for AWS S3 backups. See this issue. Thanks to @justcompile for the pull request.
FEATURE: vmbackup: store backup creation and completion time in backup_complete.ignore file of backup contents. This allows determining the exact timestamp when the backup was created and completed.
FEATURE: vmbackupmanager: add created_at field to the output of /api/v1/backups API and vmbackupmanager backup list command. See this doc for data format details.
FEATURE: vmbackupmanager: add commands for locking/unlocking backups against deletion by retention policy. See this doc for data format details.
FEATURE: vmctl: add support for different time formats for --vm-native-filter-time-start and --vm-native-filter-time-end command-line flags. See this issue.
FEATURE: vmctl: set default value for --vm-native-step-interval command-line flag to month. This enables time-based chunking of data based on monthly step value when using native migration mode. See this issue.
BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores.
BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph sum(rate(vm_slow_per_day_index_inserts_total)). See this pull request.
BUGFIX: fix possible infinite loop during indexdb rotation when -retentionTimezoneOffset command-line flag is set and the local timezone is not UTC. See this issue. Thanks to @faceair for the fix.
BUGFIX: do not panic at Windows during snapshot deletion. Instead, delete the snapshot on the next restart. See this comment for details.
BUGFIX: change the max allowed value for -memory.allowedPercent from 100 to 200. See this issue.
BUGFIX: properly limit the number of OpenTSDB HTTP concurrent requests specified via -maxConcurrentInserts command-line flag. See this issue. Thanks to @zouxiang1993 for the fix.
BUGFIX: do not ignore trailing empty field in CSV lines when importing data in CSV format. See this issue.
BUGFIX: disallow " chars when parsing Prometheus label names, since they aren't allowed by Prometheus text exposition format. Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as foo{"bar"="baz"} or {foo:"bar",baz="aaa"}. See this issue.
BUGFIX: VictoriaMetrics cluster: prevent from possible panic when the number of vmstorage nodes increases when automatic vmstorage discovery is enabled.
BUGFIX: MetricsQL: fix a panic when the duration in the query contains uppercase M suffix. Such a suffix isn't allowed to use in durations, since it clashes with a million suffix, e.g. it isn't clear whether rate(metric[5M]) means rate over 5 minutes, 5 months or 5 million seconds. See this and this issues.
BUGFIX: vmagent: properly handle the vm_promscrape_config_last_reload_successful metric after config reload. See this issue.
BUGFIX: vmagent: add __meta_kubernetes_endpoints_name label for all ports discovered from endpoint. Previously, ports not matched by Service did not have this label. See this issue for details. Thanks to @thunderbird86 for discovering and fixing the issue.
BUGFIX: vmalert: retry failed read request on the closed connection one more time. This improves rules execution reliability when connection between vmalert and datasource closes unexpectedly.
BUGFIX: vmalert: properly display an error when using query function for templating value of -external.alert.source flag. See this issue.
BUGFIX: vmalert: properly return empty slices instead of nil for /api/v1/rules and /api/v1/alerts API handlers. See this issue.
BUGFIX: vmauth: do not return invalid auth credentials in http response by default, since it may be logged by client. See this issue.
BUGFIX: vmui: fix the display of the tenant selector. See this issue.
BUGFIX: vmui: fix UI freeze when the query returns non-histogram series alongside histogram series.
BUGFIX: vmui: fix the text display on buttons in Safari 16.4.
BUGFIX: alerts-health: update threshold for TooHighMemoryUsage alert from 90% to 80%, since 90% is too high for production environments.
BUGFIX: vmbackup: fix compatibility with Windows OS. See this issue.
BUGFIX: vmctl: fix performance issue when migrating data from VictoriaMetrics according to these docs. Add the ability to speed up the data migration via --vm-native-disable-retries command-line flag. See this issue.
BUGFIX: stream aggregation: fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See this issue.

Update note: this release contains backwards-incompatible change in storage data format, so the previous versions of VictoriaMetrics will exit with the unexpected number of substrings in the part name error when trying to run them on the data created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or newer releases

SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See alpine 3.17.3 release notes.
SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See the list of issues addressed in Go1.20.3.
FEATURE: open source Graphite Render API. This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See this case study as an example.
FEATURE: release Windows binaries for single-node VictoriaMetrics, VictoriaMetrics cluster, vmbackup and vmrestore. See this, this and this issues. This release of VictoriaMetrics for Windows cannot delete snapshots due to Windows constraints. See this comment for details. This issue should be resolved in future releases.
FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds -maxLabelValueLen. This should simplify debugging for this case.
FEATURE: vmagent: show target URL when debugging target relabeling. This should simplify target relabel debugging a bit. See this pull request.
FEATURE: vmagent: add support for VictoriaMetrics remote write protocol when sending / receiving data to / from Kafka. This protocol allows saving egress network bandwidth costs when sending data from vmagent to Kafka located in another datacenter or availability zone. See this feature request.
FEATURE: vmagent: add -kafka.consumer.topic.concurrency command-line flag. It controls the number of Kafka consumer workers to use by vmagent. It should eliminate the need to start multiple vmagent instances to improve data transfer rate. See this feature request.
FEATURE: vmagent: add support for Kafka producer and consumer on arm64 machines. See this issue.
FEATURE: vmagent: delete unused buffered data at -remoteWrite.tmpDataPath directory when there is no matching -remoteWrite.url to send this data to. See this feature request.
FEATURE: vmagent: add the ability for hot reloading of stream aggregation configs. See these docs and this feature request.
FEATURE: check the contents of -relabelConfig and -streamAggr.config files additionally to -promscrape.config when single-node VictoriaMetrics runs with -dryRun command-line flag. This aligns the behaviour of single-node VictoriaMetrics with vmagent behaviour for -dryRun command-line flag.
FEATURE: vmui: automatically draw a heatmap graph when the query selects a single histogram. This simplifies analyzing histograms. See this feature request.
FEATURE: vmui: add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See this pull request.
FEATURE: vmui: hide messages longer than 3 lines in the trace. You can view the full message by clicking on the show more button. See this pull request.
FEATURE: vmui: add the ability to manually input date and time when selecting a time range. See this pull request.
FEATURE: vmui: updated usability and the search process in cardinality explorer. Made this process straightforward for user. See this pull request.
FEATURE: vmui: add the ability to collapse/expand the legend. See this pull request.
FEATURE: vmui: add tips for working with the graph and legend. See this pull request.
FEATURE: vmui: add apply and cancel buttons to settings popup. See this issue.
FEATURE: vmctl: automatically disable progress bar when TTY isn't available. See this issue.
FEATURE: vmauth: add -configCheckInterval command-line flag, which can be used for automatic re-reading the -auth.config file. See this feature request.
BUGFIX: prevent from slow snapshot creating under high data ingestion rate. See this issue.
BUGFIX: vmauth: suppress proxy protocol parsing errors in case of EOF. Usually, the error is caused by health checks and is not a sign of an actual error.
BUGFIX: vmui: fix displaying errors for each query. See this issue.
BUGFIX: vmbackup: fix snapshot not being deleted in case of error during backup. See this issue.
BUGFIX: stream aggregation: suppress series after dedup error message in logs when -remoteWrite.streamAggr.dedupInterval command-line flag is set at vmagent or when -streamAggr.dedupInterval command-line flag is set at single-node VictoriaMetrics.
BUGFIX: allow using dashes and dots in environment variables names referred in config files via %{ENV-VAR.SYNTAX}. See these docs and this issue.
BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in v1.86.0. See this issue.
BUGFIX: MetricsQL: properly convert VictoriaMetrics historgram buckets to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for the fix.
BUGFIX: vmagent: fix CPU and memory usage spikes when files pointed by file_sd_config cannot be re-read. See this_issue.
BUGFIX: prevent unexpected merges on start-up when -storage.minFreeDiskSpaceBytes is set. See the issue.
BUGFIX: properly support comma-separated filters inside retention filters. See this issue.
BUGFIX: verify response code when fetching configuration files via HTTP. See this issue.
BUGFIX: vmalert: replace empty labels with "" instead of "<no value>" during templating, as Prometheus does. See this issue.
BUGFIX: vmctl: properly pass multiple filters from --vm-native-filter-match command-line flag to the data source. Previously filters from --vm-native-filter-match were only used to discover the metric names, and the metric names like __name__="metric_name" has been taken into account, while the remaining filters were ignored. For example --vm-native-src-addr={foo="bar",baz="abc"} may found metric_name{foo="bar",baz="abc"} and filter was treated as --vm-native-src-addr={__name__="metrics_name"}, e.g. foo="bar",baz="abc" filter was ignored. See this issue.

v1.89.1

Released at 2023-03-12

BUGFIX: prevent from possible cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0.

v1.89.0

Released at 2023-03-12

Update note: this release can crash with cannot unmarshal timeseries from rollupResultCache panic after the upgrade from the previous releases. This issue can be fixed by removing caches stored on disk according to these docs. Another option is to upgrade to v1.89.1.

SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See the list of issues addressed in Go1.20.2.
FEATURE: vmctl: increase the default value for --remote-read-http-timeout command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See this pull request.
FEATURE: vmctl: migrate series one-by-one in vm-native mode. This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See this pull request and this feature request.
FEATURE: vmctl: add --vm-native-src-headers and --vm-native-dst-headers command-line flags, which can be used for setting custom HTTP headers during vm-native migration mode. Thanks to @baconmania for the pull request.
FEATURE: vmctl: add --vm-native-src-bearer-token and --vm-native-dst-bearer-token command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during vm-native migration mode. See this feature request.
FEATURE: vmctl: add --vm-native-disable-http-keep-alive command-line flag to allow vmctl to use non-persistent HTTP connections in vm-native migration mode. Thanks to @baconmania for the pull request.
FEATURE: vmalert: log number of configration files found for each specified -rule command-line flag.
FEATURE: vmalert enterprise: concurrently read config files from S3, GCS or S3-compatible object storage. This significantly improves config load speed for cases when there are thousands of files to read from the object storage.
BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See this pull request.
BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See this issue.
BUGFIX: prevent from possible invalid memory address or nil pointer dereference panic during background merge. The issue has been introduced at v1.85.0. See this issue.
BUGFIX: prevent from possible SIGBUS crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for the initial attempt to fix it.
BUGFIX: VictoriaMetrics cluster: always return is_partial: true in partial responses. Previously partial responses could be returned as non-partial in some cases.
BUGFIX: VictoriaMetrics cluster: properly take into account -rpc.disableCompression command-line flag at vmstorage. It was ignored since v1.78.0. See this pull request.
BUGFIX: vmagent: fix panic when writing data to Kafka. The panic has been introduced in v1.88.0.
BUGFIX: vmui: stop showing Please enter a valid Query and execute it error message on the first load of vmui.
BUGFIX: vmui: properly process Run in VMUI button click in VictoriaMetrics datasource plugin for Grafana.
BUGFIX: vmui: fix the display of the selected value for dropdowns on Explore page.
BUGFIX: vmui: do not send step param for instant queries. See this issue.
BUGFIX: vmauth: fix cannot serve http panic when plain HTTP request is sent to vmauth configured to accept requests over proxy protocol-encoded request (e.g. when vmauth runs with -httpListenAddr.useProxyProtocol command-line flag). The issue has been introduced at v1.87.0 when implementing this feature.
BUGFIX: vmgateway: properly parse RSA public key discovered via JWK endpoint.

v1.88.1

Released at 2023-02-27

FEATURE: add -snapshotCreateTimeout flag to allow configuring timeout for snapshot process. See this issue.
FEATURE: expose vm_http_requests_total and vm_http_request_errors_total metrics for snapshot/* paths at VictoriaMetrics cluster vmstorage and VictoriaMetrics Single. See this issue.
FEATURE: vmgateway: add the ability to discover keys for JWT verification via OpenID discovery endpoint. See these docs.
FEATURE: add -internStringDisableCache command-line flag for disabling the cache for interned strings. This flag may be useful in some cases for reducing memory usage at the cost of higher CPU usage.
FEATURE: add -internStringCacheExpireDuration command-line flag for controlling the lifetime of cached interned strings.
BUGFIX: MetricsQL: fix panic when executing the query aggr_func(rollup*(some_value)). The panic has been introduced in v1.88.0.
BUGFIX: vmagent: use the provided -remoteWrite.* auth options when determining whether the remote storage supports VictoriaMetrics remote write protocol. Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol.
BUGFIX: vmagent: do not register vm_promscrape_config_* metrics if -promscrape.config flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts.
BUGFIX: vmctl: skip measurements with no fields when migrating data from influxdb. See this issue.
BUGFIX: delete failed snapshot contents from disk on failed attempt to create snapshot. Previously failed snapshot contents could remain on disk in incomplete state. See this issue

v1.88.0

Released at 2023-02-24

SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See alpine 3.17.2 release notes.
SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See the list of issues addressed in Go1.20.1.
FEATURE: vmagent: add support for VictoriaMetrics remote write protocol. This protocol allows saving egress network bandwidth costs when sending data from vmagent to VictoriaMetrics located in another datacenter or availability zone. This also allows reducing disk IO under high load when vmagent starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See this feature request.
FEATURE: vmagent: add support for Kuma Control Plane targets discovery aka kuma_sd_configs. See this issue.
FEATURE: vmgateway: add the ability to verify JWT signature via JWKS endpoint. See these docs.
FEATURE: vmauth: add the ability to limit the number of concurrent requests on a per-user basis via -maxConcurrentPerUserRequests command-line flag and via max_concurrent_requests config option. See this feature request and these docs.
FEATURE: vmauth: automatically retry failing GET requests on all the configured backends. Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends.
FEATURE: vmauth: choose the backend with the minimum number of concurrently executed requests among the configured backends in a round-robin manner for serving the incoming requests. This allows spreading the load among backends more evenly, while improving the response time.
FEATURE: vmalert enterprise: add ability to read alerting and recording rules from S3, GCS or S3-compatible object storage. See these docs.
FEATURE: vmctl: automatically retry requests to remote storage if up to 5 errors occur during the data migration process. This should help continuing the data migration process on temporary errors. Previously vmctl was stopping after the first error. See this feature request.
FEATURE: MetricsQL: support optional 2nd argument min, max or avg for rollup, rollup_delta, rollup_deriv, rollup_increase, rollup_rate and rollup_scrape_interval function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, rollup_rate(requests_total[1i], "max") would return only the max increase rates for requests_total metric per each interval between adjacent points on the graph. See this article for details.
FEATURE: MetricsQL: support optional 2nd argument open, low, high, close for rollup_candlestick function. If the second argument is passed, then the function returns only the selected aggregation type.
FEATURE: MetricsQL: add share(q) aggregate function.
FEATURE: MetricsQL: add mad_over_time(m[d]) function for calculating the median absolute deviation over raw samples on the lookbehind window d. See this feature request.
FEATURE: MetricsQL: add range_mad(q) function for calculating the median absolute deviation over points per each time series returned by q.
FEATURE: MetricsQL: add range_zscore(q) function for calculating z-score over points per each time series returned from q.
FEATURE: MetricsQL: add range_trim_outliers(k, q) function for dropping outliers located farther than k*range_mad(q) from the range_median(q). This should help removing outliers during query time at this issue.
FEATURE: MetricsQL: add range_trim_zscore(z, q) function for dropping outliers located farther than z*range_stddev(q) from range_avg(q). This should help removing outliers during query time at this issue.
FEATURE: vmui: show median instead of avg in graph tooltip and line legend, since median is more tolerant against spikes. See this issue.
FEATURE: add -search.maxSeriesPerAggrFunc command-line flag, which can be used for limiting the number of time series MetricsQL aggregate functions can return in a single query. This flag can be useful for preventing OOMs when count_values function is improperly used.
FEATURE: vmui: small UX improvements for mobile view. See this feature request and this pull request.
FEATURE: add -search.logQueryMemoryUsage command-line flag for logging queries, which need more memory than specified by this command-line flag. See this feature request. Thanks to @michal-kralik for the idea and the intial implementation.
FEATURE: allow setting zero value for -search.latencyOffset command-line flag. This may be needed in some cases. Previously the minimum supported value for -search.latencyOffset command-line flag was 1s.
BUGFIX: vmagent: immediately cancel in-flight scrape requests during configuration reload when stream parsing mode is disabled. Previously vmagent could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.
BUGFIX: vmagent: do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes http keep-alive connection between scrapes. See this and this issues.
BUGFIX: vmagent: fix Azure service discovery inside Azure Container App. See this issue. Thanks to @MattiasAng for the fix!
BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from cannot create hard links from ...must-remove... errors when making snapshots / backups. See this issue.
BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during background merges of big parts on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in v1.85.0 when implementing this feature. See also this issue.
BUGFIX: properly parse timestamps in milliseconds when ingesting data via OpenTSDB telnet put protocol. Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the pull request.
BUGFIX: MetricsQL: do not add extrapolated points outside the real points when using interpolate() function. See this issue.

v1.87.6

Released at 2023-05-18

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See the list of issues addressed in Go1.20.4.
SECURITY: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See alpine 3.18.0 release notes.
SECURITY: serve /robots.txt content to disallow indexing of the exposed instances by search engines. See this issue for details.
BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores.
BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph sum(rate(vm_slow_per_day_index_inserts_total)). See this pull request.
BUGFIX: do not ignore trailing empty field in CSV lines when importing data in CSV format. See this issue.
BUGFIX: disallow " chars when parsing Prometheus label names, since they aren't allowed by Prometheus text exposition format. Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as foo{"bar"="baz"} or {foo:"bar",baz="aaa"}. See this issue.
BUGFIX: MetricsQL: fix a panic when the duration in the query contains uppercase M suffix. Such a suffix isn't allowed to use in durations, since it clashes with a million suffix, e.g. it isn't clear whether rate(metric[5M]) means rate over 5 minutes, 5 months or 5 million seconds. See this and this issues.
BUGFIX: VictoriaMetrics cluster: prevent from possible panic when the number of vmstorage nodes increases when automatic vmstorage discovery is enabled.
BUGFIX: properly limit the number of OpenTSDB HTTP concurrent requests specified via -maxConcurrentInserts command-line flag. See this issue. Thanks to @zouxiang1993 for the fix.
BUGFIX: vmalert: properly return empty slices instead of nil for /api/v1/rules and /api/v1/alerts API handlers. See this issue.
BUGFIX: vmagent: add __meta_kubernetes_endpoints_name label for all ports discovered from endpoint. Previously, ports not matched by Service did not have this label. See this issue for details. Thanks to @thunderbird86 for discovering and fixing the issue.
BUGFIX: fix possible infinite loop during indexdb rotation when -retentionTimezoneOffset command-line flag is set and the local timezone is not UTC. See this issue. Thanks to @faceair for the fix.
BUGFIX: vmauth: do not return invalid auth credentials in http response by default, since it may be logged by client. See this issue.
BUGFIX: alerts-health: update threshold for TooHighMemoryUsage alert from 90% to 80%, since 90% is too high for production environments.
BUGFIX: vmagent: properly handle the vm_promscrape_config_last_reload_successful metric after config reload. See this issue.
BUGFIX: stream aggregation: fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See this issue.
BUGFIX: stream aggregation: suppress series after dedup error message in logs when -remoteWrite.streamAggr.dedupInterval command-line flag is set at vmagent or when -streamAggr.dedupInterval command-line flag is set at single-node VictoriaMetrics.

v1.87.5

Released at 2023-04-06

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See alpine 3.17.3 release notes.
SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See the list of issues addressed in Go1.20.3.
BUGFIX: MetricsQL: properly convert VictoriaMetrics historgram buckets to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for the fix.
BUGFIX: vmagent: fix CPU and memory usage spikes when files pointed by file_sd_config cannot be re-read. See this_issue.
BUGFIX: prevent unexpected merges on start-up when -storage.minFreeDiskSpaceBytes is set. See the issue.
BUGFIX: properly support comma-separated filters inside retention filters. See this issue.
BUGFIX: verify response code when fetching configuration files via HTTP. See this issue.

v1.87.4

Released at 2023-03-25

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

BUGFIX: prevent from slow snapshot creating under high data ingestion rate. See this issue.
BUGFIX: vmauth: suppress proxy protocol parsing errors in case of EOF. Usually, the error is caused by health checks and is not a sign of an actual error.
BUGFIX: vmbackup: fix snapshot not being deleted in case of error during backup. See this issue.
BUGFIX: allow using dashes and dots in environment variables names referred in config files via %{ENV-VAR.SYNTAX}. See these docs and this issue.
BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in v1.86.0. See this issue.

v1.87.3

Released at 2023-03-12

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See the list of issues addressed in Go1.20.2.
BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See this pull request.
BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See this issue.
BUGFIX: prevent from possible invalid memory address or nil pointer dereference panic during background merge. The issue has been introduced at v1.85.0. See this issue.
BUGFIX: prevent from possible SIGBUS crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for the initial attempt to fix it.
BUGFIX: VictoriaMetrics cluster: always return is_partial: true in partial responses. Previously partial responses could be returned as non-partial in some cases.
BUGFIX: VictoriaMetrics cluster: properly take into account -rpc.disableCompression command-line flag at vmstorage. It was ignored since v1.78.0. See this pull request.
BUGFIX: vmagent: do not register vm_promscrape_config_* metrics if -promscrape.config flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts.
BUGFIX: vmctl: skip measurements with no fields when migrating data from influxdb. See this issue.
BUGFIX: vmauth: fix cannot serve http panic when plain HTTP request is sent to vmauth configured to accept requests over proxy protocol-encoded request (e.g. when vmauth runs with -httpListenAddr.useProxyProtocol command-line flag). The issue has been introduced at v1.87.0 when implementing this feature.

v1.87.2

Released at 2023-02-24

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See alpine 3.17.2 release notes.
SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See the list of issues addressed in Go1.20.1.
BUGFIX: vmagent: immediately cancel in-flight scrape requests during configuration reload when stream parsing mode is disabled. Previously vmagent could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.
BUGFIX: vmagent: do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes http keep-alive connection between scrapes. See this and this issues.
BUGFIX: vmagent: fix Azure service discovery inside Azure Container App. See this issue. Thanks to @MattiasAng for the fix!
BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from cannot create hard links from ...must-remove... errors when making snapshots / backups. See this issue.
BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during background merges of big parts on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in v1.85.0 when implementing this feature. See also this issue.
BUGFIX: properly parse timestamps in milliseconds when ingesting data via OpenTSDB telnet put protocol. Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the pull request.
BUGFIX: MetricsQL: do not add extrapolated points outside the real points when using interpolate() function. See this issue.

v1.87.1

Released at 2023-02-09

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

FEATURE: vmalert: alerts state restore procedure was changed to become asynchronous. It doesn't block groups start anymore which significantly improves vmalert's startup time. This also means that -remoteRead.ignoreRestoreErrors command-line flag becomes deprecated now and will have no effect if configured. While previously state restore attempt was made for all the loaded alerting rules, now it is called only for alerts which became active after the first evaluation. See this issue.
FEATURE: vmui: optimize VMUI for use from smartphones and tablets. See this feature request.
FEATURE: vmui: add ability to search tenants in the drop-down list for the tenant selector. See this feature request.
FEATURE: vmui: add avg/min/max/last values to line legends and tooltips for graphs. See this feature request.
FEATURE: vmui: hide the default per-job resource usage dashboard if there is a custom dashboard exists at the directory specified via -vmui.customDashboardsPath command-line flag. See this feature request.
BUGFIX: vmagent: fix panic in HashiCorp Nomad service discovery. Thanks to @mr-karan for the pull request.
BUGFIX: vmalert: fix display of rules number per-group for groups with identical names in UI.
BUGFIX: vmalert: prevent disabling state updates tracking per rule via setting values < 1. The minimum number of update states to track is now set to 1.
BUGFIX: vmalert: properly update debug and update_entries_limit rule's params on config's hot-reload.
BUGFIX: properly initialize the vm_concurrent_insert_current metric before exposing it. Previously this metric could be left uninitialized in some cases, e.g. its value was zero. This could lead to false alerts for the query avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity. See this issue.
BUGFIX: vmagent: immediately cancel in-flight scrape requests during configuration reload when using stream parsing mode. Previously vmagent could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.
BUGFIX: vmgateway: do not validate JWT signature if no public keys are provided. Previously this could result in the error setting up jwt verification error.

v1.87.0

Released at 2023-02-01

v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

FEATURE: stream aggregation: add the ability to de-duplicate input samples before aggregation via -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval command-line options.
FEATURE: vmui: add dark mode - it can be selected via settings menu in the top right corner. See this pull request.
FEATURE: vmui: improve visual appearance of the top menu. See this feature request.
FEATURE: vmui: embed fonts into binary instead of loading them from external sources. This allows using vmui in full from isolated networks without access to Internet. Thanks to @ScottKevill for the pull request.
FEATURE: vmui: add ability to switch between tenants by selecting the needed tenant in the drop-down list at the top right corner of the UI. See this pull request.
FEATURE: vmagent: reduce memory usage when sending stale markers for targets, which expose big number of metrics. See this and this issues.
FEATURE: vmagent: add __meta_kubernetes_pod_container_id meta-label to the targets discovered via kubernetes_sd_configs. This label has been added in Prometheus starting from v2.42.0. See this feature request.
FEATURE: vmagent: add __meta_azure_machine_size meta-label to the targets discovered via azure_sd_configs. This label has been added in Prometheus starting from v2.42.0. See this pull request.
FEATURE: vmauth: allow limiting the number of concurrent requests sent to vmauth via -maxConcurrentRequests command-line flag. This allows controlling memory usage of vmauth and the resource usage of backends behind vmauth. See this feature request. Thanks to @dmitryk-dk for the initial implementation.
FEATURE: allow using VictoriaMetrics components behind proxies, which communicate with the backend via proxy protocol. See this feature request. For example, vmauth accepts proxy protocol connections when it starts with -httpListenAddr.useProxyProtocol command-line flag.
FEATURE: add -internStringMaxLen command-line flag, which can be used for fine-tuning RAM vs CPU usage in certain workloads. For example, if the stored time series contain long labels, then it may be useful reducing the -internStringMaxLen in order to reduce memory usage at the cost of increased CPU usage. See this issue.
FEATURE: provide GOARCH=386 binaries for single-node VictoriaMetrics, vmagent, vmalert, vmauth, vmbackup and vmrestore components at releases page. See this feature request. Thanks to @denisgolius for the pull request.
BUGFIX: fix a bug, which could prevent background merges for the previous partitions until restart if the storage didn't have enough disk space for final deduplication and down-sampling.
BUGFIX: fix a bug, which could lead to increased CPU usage and disk IO usage when adding data to previous months and when the deduplication or downsampling is enabled. See this pull request.
BUGFIX: VictoriaMetrics cluster: propagate all the timeout-related errors from vmstorage to vmselect. Previously some timeout errors weren't returned from vmselect to vmstorage. Instead, vmstorage could log the error and close the connection to vmselect, so vmselect was logging cryptic errors such as cannot execute funcName="..." on vmstorage "...": EOF.
BUGFIX: vmui: add support for time zone selection for older versions of browsers. See this pull request.
BUGFIX: vmagent: update API version for ec2_sd_configs to fix the issue with missing __meta_ec2_availability_zone_id attribute.
BUGFIX: vmagent: properly return 200 OK HTTP status code when importing data via Pushgateway protocol. See this issue.
BUGFIX: vmagent: do not add exported_ prefix to scraped metric names, which clash with the automatically generated metric names if honor_labels: true option is set in the scrape_config. See the this and this issues.
BUGFIX: vmauth: allow re-entering authorization info in the web browser if the entered info was incorrect. Previously it was non-trivial to do via the web browser, since vmauth was returning 400 Bad Request instead of 401 Unauthorized http response code.
BUGFIX: vmauth: always log the client address and the requested URL on proxying errors. Previously some errors could miss this information.
BUGFIX: vmbackup: fix snapshot not being deleted after backup completion. This issue could result in unnecessary snapshots being stored, it is required to delete unnecessary snapshots manually. See the this issue.
BUGFIX: VictoriaMetrics cluster: fix panic on top-level vmselect nodes of multi-level setup when the -replicationFactor flag is set and request contains trace query parameter. See this issue.

v1.86.2

Released at 2023-01-18

SECURITY: vmbackup: do not expose basic auth passwords from -snapshot.createURL and -snapshot.deleteURL command-line flags in logs. Thanks to @toanju for the pull request.
FEATURE: vmui: add ability to show custom dashboards at vmui by specifying a path to a directory with dashboard config files via -vmui.customDashboardsPath command-line flag. See this feature request and these docs.
FEATURE: vmui: apply the step globally to all the displayed graphs. See this feature request.
FEATURE: vmui: improve the appearance of graph lines by using more visually distinct colors. See this feature request.
BUGFIX: do not slow down concurrently executed queries during assisted merges, since assisted merges already prioritize data ingestion over queries. The probability of assisted merges has been increased starting from v1.85.0 because of internal refactoring. This could result in slowed down queries when there is a plenty of free CPU resources. See this and this issues.
BUGFIX: reduce the increased CPU usage at vmselect to v1.85.3 level when processing heavy queries. See this issue.
BUGFIX: retention filters: fix FATAL: cannot locate metric name for metricID=...: EOF panic, which could occur when retention filters are enabled.
BUGFIX: vmagent: properly cancel in-flight service discovery requests for consul_sd_configs and nomad_sd_configs when the service list changes. See this issue.
BUGFIX: vmagent: dockerswarm_sd_configs: apply filters only to objects of the specified role. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See this issue.
BUGFIX: vmagent: suppress all the scrape errors when -promscrape.suppressScrapeErrors is enabled. Previously some scrape errors were logged even if -promscrape.suppressScrapeErrors flag was set.
BUGFIX: vmagent: consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information.
BUGFIX: vmagent: do not send stale markers to remote storage for series exceeding the configured series limit. See this issue.
BUGFIX: vmagent: properly apply series limit when staleness tracking is disabled.
BUGFIX: vmagent: reduce memory usage spikes when big number of scrape targets disappear at once. See this issue. Thanks to @lzfhust for the initial fix.
BUGFIX: Pushgateway import: properly return 200 OK HTTP response code. See this issue.
BUGFIX: MetricsQL: properly parse M and Mi suffixes as 1e6 multipliers in 1M and 1Mi numeric constants. See this issue. The issue has been introduced in v1.86.0.
BUGFIX: vmui: properly display range query results at Table view. For example, up[5m] query now shows all the raw samples for the last 5 minutes for the up metric at the Table view. See this issue.

v1.86.1

Released at 2023-01-10

BUGFIX: return correct query results over time series with gaps. The issue has been introduced in v1.86.0.
BUGFIX: properly take into account the timeout passed by vmselect to vmstorage during query execution. This issue could result in the following error logs at vmstorage under load: cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed. The issue has been introduced in v1.86.0.

v1.86.0

Released at 2023-01-10

It is recommended upgrading to VictoriaMetrics v1.86.1 because v1.86.0 contains a bug, which could lead to incorrect query results over time series with gaps.

Update note 1: This release changes the logic behind -maxConcurrentInserts command-line flag. Previously this flag was limiting the number of concurrent connections established from clients, which send data to VictoriaMetrics. Some of these connections could be temporarily idle. Such connections do not take significant CPU and memory resources, so there is no need in limiting their count. The new logic takes into account only those connections, which actively ingest new data to VictoriaMetrics and to vmagent. This means that the default -maxConcurrentInserts value should handle cases, which could require increasing the value in the previous releases. So it is recommended trying to remove the explicitly set -maxConcurrentInserts command-line flag after upgrading to this release and verifying whether this reduces CPU and memory usage.

Update note 2: The vm_concurrent_addrows_current and vm_concurrent_addrows_capacity metrics exported by vmstorage are replaced with vm_concurrent_insert_current and vm_concurrent_insert_capacity metrics in order to be consistent with the corresponding metrics exported by vminsert. Please update queries in dahsboards and alerting rules with new metric names if old metric names are used there.

FEATURE: vmagent: add support for aggregation of incoming samples by time and by labels. See these docs and this feature request.
FEATURE: vmagent: reduce memory usage when scraping big number of targets without the need to enable stream parsing mode.
FEATURE: vmagent: add support for Prometheus-compatible target discovery for HashiCorp Nomad services via nomad_sd_configs. See this feature request. Thanks to @mr-karan for the implementation.
FEATURE: vmagent: automatically pre-fetch metric_relabel_configs and the target labels when clicking on the debug metrics relabeling link at the http://vmagent:8429/targets page at the particular target. See these docs.
FEATURE: vmui: add ability to explore metrics exported by a particular job / instance. See these docs and this feature request.
FEATURE: allow passing partial RFC3339 date/time to time, start and end query args at querying APIs and export APIs. For example, 2022 is equivalent to 2022-01-01T00:00:00Z, while 2022-01-30T14 is equivalent to 2022-01-30T14:00:00Z. See these docs.
FEATURE: MetricsQL: allow using unicode letters in identifiers. For example, температура{город="Киев"} is a valid MetricsQL expression now. Previously every non-ascii letters should be escaped with \ char when used inside MetricsQL expression: \т\е\м\п\е\р\а\т\у\р\а{\г\о\р\о\д="Киев"}. Now both expressions are equivalent. Thanks to @hzwwww for the pull request.
FEATURE: relabeling: add support for keepequal and dropequal relabeling actions, which are supported by Prometheus starting from v2.41.0. These relabeling actions are almost identical to keep_if_equal and drop_if_equal relabeling actions supported by VictoriaMetrics since v1.38.0 - see these docs - so it is recommended sticking to keep_if_equal and drop_if_equal actions instead of switching to keepequal and dropequal.
FEATURE: csvimport: support empty values for imported metrics. See this issue.
FEATURE: vmalert: allow configuring the default number of stored rule's update states in memory via global -rule.updateEntriesLimit command-line flag or per-rule via rule's update_entries_limit configuration param. See these docs and this pull request.
FEATURE: improve the logic benhind -maxConcurrentInserts command-line flag. Previously this flag was limiting the number of concurrent connections from clients, which write data to VictoriaMetrics or vmagent. Some of these connections could be idle for some time. These connections do not need significant amounts of CPU and memory, so there is no sense in limiting their count. The updated logic behind -maxConcurrentInserts limits the number of active insert requests, not counting idle connections.
FEATURE: protect all the http endpoints with -httpAuth.* command-line flag. Previously endpoints protected by -*AuthKey command-line flags weren't protected by -httpAuth.*. This could complicate the proper security setup. See this issue.
FEATURE: VictoriaMetrics cluster: add -maxConcurrentInserts and -insert.maxQueueDuration command-line flags to vmstorage, so they could be tuned if needed in the same way as at vminsert nodes.
FEATURE: VictoriaMetrics cluster: limit the number of concurrently executed requests at vmstorage proportionally to the number of available CPU cores, since every request can saturate a single CPU core at vmstorage. Previously a single vmstorage could accept and start processing arbitrary number of concurrent requests received from big number of vmselect nodes. This could result in increased RAM, CPU and disk IO usage or event to out of memory crash at vmstorage side under high load. The limit can be fine-tuned if needed via -search.maxConcurrentRequests command-line flag at vmstorage according to these docs. vmstorage now exposes the following additional metrics at http://vmstorage:8482/metrics page:
- vm_vmselect_concurrent_requests_capacity - the maximum number of requests allowed to execute concurrently
- vm_vmselect_concurrent_requests_current - the current number of concurrently executed requests
- vm_vmselect_concurrent_requests_limit_reached_total - the total number of requests, which were put in the wait queue when -search.maxConcurrentRequests concurrent requests are being executed
- vm_vmselect_concurrent_requests_limit_timeout_total - the total number of canceled requests because they were sitting in the wait queue for more than -search.maxQueueDuration
BUGFIX: vmui: properly update the step value in url after the step input field has been manually changed. This allows preserving the proper step when copy-n-pasting the url to another instance of web browser. See this issue.
BUGFIX: vmui: properly update tooltip when quickly hovering multiple lines on the graph. See this issue.
BUGFIX: properly parse floating-point numbers without integer or fractional parts such as .123 and 20. during data import. See this issue.
BUGFIX: MetricsQL: properly parse durations with uppercase suffixes such as 10S, 5MS, 1W, etc. See this issue.
BUGFIX: vmagent: fix a panic during target discovery when vmagent runs with -promscrape.dropOriginalLabels command-line flag. See this issue. The bug has been introduced in v1.85.0.
BUGFIX: vmagent: dockerswarm_sd_configs: properly encode filters field. See this issue.
BUGFIX: vmagent: fix possible resource leak after hot reload of the updated consul_sd_configs. See this issue.
BUGFIX: vmagent: fix a panic in gce_sd_configs when the discovered instance has zero labels. See this issue. The issue has been introduced in v1.85.0.
BUGFIX: properly return label names starting from uppercase such as CamelCaseLabel from /api/v1/labels. See this issue.
BUGFIX: fix opentsdb HTTP endpoint not respecting -httpAuth.* flags. See this issue
BUGFIX: consistently select the sample with the biggest value out of samples with identical timestamps during querying when the deduplication is enabled according to this feature request. Previously random samples could be selected during querying.

v1.85.3

Released at 2022-12-20

Update note 1: This and newer releases of VictoriaMetrics may return gaps for rate(m[d]) queries on short time ranges if [d] lookbehind window is set expliticly. For example, rate(http_requests_total[$__interval]). This reduces confusion level when the user expects the needed results from the query with explicitly set lookbehind window. See this issue. The previous gap filling behaviour can be restored by removing explicit lookbehind window [d] from the query, e.g. by substituting the rate(m[d]) with rate(m). See these docs for details.

BUGFIX: fix error when searching for TSIDs by metricIDs in the previous indexdb: EOF error, which can occur during queries after unclean shutdown of VictoriaMetrics (e.g. via hardware reset, out of memory crash or kill -9). The error has been introduced in v1.85.2. See this issue.
BUGFIX: VictoriaMetrics enterprise: expose proper values for vm_downsampling_partitions_scheduled and vm_downsampling_partitions_scheduled_size_bytes metrics, which were added at v1.78.0. See this feature request.
BUGFIX: MetricsQL: never extend explicitly set lookbehind window for rate() function. This reduces the level of confusion when the user expects the needed results after explicitly seting the lookbehind window [d] in the query rate(m[d]). Previously VictoriaMetrics could silently extend the lookbehind window, so it covers at least two raw samples. Now this behavior works only if the lookbehind window in square brackets isn't set explicitly, e.g. in the case of rate(m). See this issue for details.
BUGFIX: vmagent: respect -usePromCompatibleNaming flag if no relabeling or extra labels were set. See this issue for details.
BUGFIX: vmui: fix the wrong legend when queries are hidden. See this issue.
BUGFIX: vmui: fix incorrect time selection after the timezone change. See this pull request.

v1.85.2

Released at 2022-12-19

FEATURE: support overriding of -search.latencyOffset value via URL param latency_offset when performing requests to /api/v1/query and /api/v1/query_range. See this issue.
FEATURE: allow changing field names in JSON logs if VictoriaMetrics components are started with -loggerFormat=json command-line flags. The field names can be changed with the -loggerJSONFields command-line flag. For example -loggerJSONFields=ts:timestamp,msg:message would rename ts and msg fields on the output JSON to timestamp and message fields. See this feature request. Thanks to @michal-kralik for the pull request.
FEATURE: vmagent: expose __meta_consul_tag_<tagname> and __meta_consul_tagpresent_<tagname> labels for targets discovered via consul_sd_configs. This simplifies converting Consul service tags to target labels with a simple relabeling rule:
```
- action: labelmap
  regex: __meta_consul_tag_(.+)
```
This resolves this StackOverflow question.
BUGFIX: properly return query results for time series, which stop receiving new samples after the rotation of indexdb. Previously such time series could be missing in query results. See this issue. The issue has been introduced in v1.83.0.
BUGFIX: allow specifying values bigger than 2GiB to the following command-line flag values on 32-bit architectures (386 and arm): -storage.minFreeDiskSpaceBytes and -remoteWrite.maxDiskUsagePerURL. Previously values bigger than 2GiB were incorrectly truncated on these architectures.
BUGFIX: vmagent: stop dropping metric name by a mistake on the /metric-relabel-debug page.

v1.85.1

Released at 2022-12-14

It is recommended upgrading to VictoriaMetrics v1.85.2 because of the bug, which may result in incomplete query results for historical time series.

FEATURE: vmalert: support $for or .For template variables in alert's annotations. See this issue.
BUGFIX: DataDog protocol parser: do not re-use host and device fields from the previously parsed messages if these fields are missing in the currently parsed message. See this issue.
BUGFIX: reduce CPU usage when the regex-based relabeling rules are applied to more than 100K unique Graphite metrics. See this issue. The issue was introduced in v1.82.0.
BUGFIX: do not block merges of small parts by merges of big parts on hosts with small number of CPU cores. This issue could result in the increasing number of storage/small parts while big merge is in progress. This, in turn, could result in increased CPU usage and memory usage during querying, since queries need to inspect bigger number of small parts. The issue has been introduced in v1.85.0.
BUGFIX: vmbackup: fix the The source request body for synchronous copy is too large and exceeds the maximum permissible limit (256MB) error when performing backups to Azure blob storage. See this issue.

v1.85.0

Released at 2022-12-11

It is recommended upgrading to VictoriaMetrics v1.85.2 because of the bug, which may result in incomplete query results for historical time series.

Update note 1: this release drops support for direct upgrade from VictoriaMetrics versions prior v1.28.0. Please upgrade to v1.84.0, wait until finished round 2 of background conversion line is emitted to log by single-node VictoriaMetrics or by vmstorage, and then upgrade to newer releases.

Update note 2: this release splits type="indexdb" metrics into type="indexdb/inmemory" and type="indexdb/file" metrics. This may break old dashboards and alerting rules, which contain label filter on {type="indexdb"}. Such label filter must be substituted with {type=~"indexdb.*"}, so it matches indexdb from the previous releases and indexdb/inmemory + indexdb/file from new releases. It is recommended upgrading to the latest available dashboards and alerting rules mentioned in these docs, since they already contain fixed label filters.

Update note 3: this release deprecates relabel_debug and metric_relabel_debug config options in scrape_configs. The -relabelDebug, -remoteWrite.relabelDebug and -remoteWrite.urlRelabelDebug command-line options are also deprecated. Use more powerful target-level relabel debugging and metric-level relabel debugging instead as documented here.

FEATURE: vmagent: provide enhanced target-level and metric-level relabel debugging. See these docs and this issue.
FEATURE: leave a sample with the biggest value for identical timestamps per each -dedup.minScrapeInterval discrete interval when the deduplication is enabled. See this issue.
FEATURE: add -inmemoryDataFlushInterval command-line flag, which can be used for controlling the frequency of in-memory data flush to disk. The data flush frequency can be reduced when VictoriaMetrics stores data to low-end flash device with limited number of write cycles (for example, on Raspberry PI). See this feature request.
FEATURE: expose additional metrics for indexdb and storage parts stored in memory and for indexdb parts stored in files (see storage docs for technical details):
- vm_active_merges{type="storage/inmemory"} - active merges for in-memory storage parts
- vm_active_merges{type="indexdb/inmemory"} - active merges for in-memory indexdb parts
- vm_active_merges{type="indexdb/file"} - active merges for file-based indexdb parts
- vm_merges_total{type="storage/inmemory"} - the total merges for in-memory storage parts
- vm_merges_total{type="indexdb/inmemory"} - the total merges for in-memory indexdb parts
- vm_merges_total{type="indexdb/file"} - the total merges for file-based indexdb parts
- vm_rows_merged_total{type="storage/inmemory"} - the total rows merged for in-memory storage parts
- vm_rows_merged_total{type="indexdb/inmemory"} - the total rows merged for in-memory indexdb parts
- vm_rows_merged_total{type="indexdb/file"} - the total rows merged for file-based indexdb parts
- vm_rows_deleted_total{type="storage/inmemory"} - the total rows deleted for in-memory storage parts
- vm_assisted_merges_total{type="storage/inmemory"} - the total number of assisted merges for in-memory storage parts
- vm_assisted_merges_total{type="indexdb/inmemory"} - the total number of assisted merges for in-memory indexdb parts
- vm_parts{type="storage/inmemory"} - the total number of in-memory storage parts
- vm_parts{type="indexdb/inmemory"} - the total number of in-memory indexdb parts
- vm_parts{type="indexdb/file"} - the total number of file-based indexdb parts
- vm_blocks{type="storage/inmemory"} - the total number of in-memory storage blocks
- vm_blocks{type="indexdb/inmemory"} - the total number of in-memory indexdb blocks
- vm_blocks{type="indexdb/file"} - the total number of file-based indexdb blocks
- vm_data_size_bytes{type="storage/inmemory"} - the total size of in-memory storage blocks
- vm_data_size_bytes{type="indexdb/inmemory"} - the total size of in-memory indexdb blocks
- vm_data_size_bytes{type="indexdb/file"} - the total size of file-based indexdb blocks
- vm_rows{type="storage/inmemory"} - the total number of in-memory storage rows
- vm_rows{type="indexdb/inmemory"} - the total number of in-memory indexdb rows
- vm_rows{type="indexdb/file"} - the total number of file-based indexdb rows
FEATURE: DataDog parser: add device tag when it is passed in the device field is present in the series object of the input request. Thanks to @PerGon for the provided pull request.
FEATURE: vmagent: improve service discovery performance when discovering big number of targets (10K and more).
FEATURE: vmagent: allow using series_limit option for limiting the number of series a single scrape target generates in stream parsing mode. See this feature request.
FEATURE: vmagent: allow using sample_limit option for limiting the number of metrics a single scrape target can expose in every response sent over stream parsing mode.
FEATURE: vmagent: add exported_ prefix to metric names exported by scrape targets if these metric names clash with automatically generated metrics such as up, scrape_samples_scraped, etc. This prevents from corruption of automatically generated metrics. See this issue.
FEATURE: vmagent: make the host label optional in DataDog data ingestion protocol. See this issue.
FEATURE: VictoriaMetrics cluster: improve error message when the requested path cannot be properly parsed, so users could identify the issue and properly fix the path. Now the error message links to url format docs. See this issue.
FEATURE: VictoriaMetrics enterprise cluster: add -storageNode.discoveryInterval command-line flag to vmselect and vminsert to control load on DNS servers when automatic discovery of vmstorage nodes is enabled. See this issue.
FEATURE: VictoriaMetrics enterprise cluster: allow reading and updating the list of vmstorage nodes at vmselect and vminsert nodes via file. See automatic discovery of vmstorage for details.
FEATURE: vmalert: reduce memory and CPU usage by up to 50% on setups with thousands of recording/alerting groups. See this issue.
FEATURE: vmalert: add -remoteWrite.sendTimeout command-line flag, which allows configuring timeout for sending data to -remoteWrite.url. See this issue.
FEATURE: vmctl: add ability to migrate data between VictoriaMetrics clusters with automatic tenants discovery. See these docs and this issue.
FEATURE: vmctl: add ability to copy data from sources via Prometheus remote_read protocol. See these docs. The related issues: one and two.
FEATURE: vmui: allow changing timezones for the requested data. See this issue.
FEATURE: vmui: provide fast path for hiding results for all the queries except the given one by clicking eye icon with ctrl key pressed. See this feature request.
FEATURE: MetricsQL: add range_trim_spikes(phi, q) function for trimming phi percent of the largest spikes per each time series returned by q. See these docs.
FEATURE: MetricsQL: allow passing inf arg into limitk, topk, bottomk and other functions, which accept numeric arg, which limits the number of output time series. See this feature request.
FEATURE: vmgateway: add support for JWT token signature verification. See these docs for details.
FEATURE: put the version of VictoriaMetrics in the first message of query trace. This should simplify debugging.
BUGFIX: vmagent: fix the The request did not have a subscription or a valid tenant level resource provider error when discovering Azure targets with azure_sd_configs. See this issue.
BUGFIX: vmalert: properly pass HTTP headers during the alert state restore procedure. See this issue.
BUGFIX: vmalert: properly specify rule evaluation step during the replay mode. The step value was previously overriden by -datasource.queryStep command-line flag.
BUGFIX: vmalert: properly return the error message from remote-write failures. Before, error was ignored and only vmalert_remotewrite_errors_total was incremented.
BUGFIX: vmui: fix sticky tooltip sizing, which could prevent from closing the tooltip. See this issue.
BUGFIX: vmui: properly put multi-line queries in the url, so it could be copy-n-pasted and opened without issues in a new browser tab. Previously the url for multi-line query couldn't be opened. See this issue.
BUGFIX: vmui: correctly handle up and down keypresses when editing multi-line queries. See this issue.

v1.84.0

Released at 2022-11-25

It is recommended upgrading to VictoriaMetrics v1.85.2 because of the bug, which may result in incomplete query results for historical time series.

FEATURE: add support for Pushgateway data import format via /api/v1/import/prometheus url. See these docs and this issue. Thanks to @PerGon for the intial implementation.
FEATURE: VictoriaMetrics cluster: add http://<vmselect>:8481/admin/tenants API endpoint for returning a list of registered tenants. See these docs for details.
FEATURE: VictoriaMetrics enterprise: add -storageNode.filter command-line flag for filtering the discovered vmstorage nodes with arbitrary regular expressions. See this feature request.
FEATURE: MetricsQL: allow using numeric values with K, Ki, M, Mi, G, Gi, T and Ti suffixes inside MetricsQL queries. For example 8Ki equals to 8*1024, while 8.2M equals to 8.2*1000*1000.
FEATURE: MetricsQL: add range_normalize function for normalizing multiple time series into [0...1] value range. This function is useful for correlation analysis of time series with distinct value ranges. See this issue.
FEATURE: MetricsQL: add range_linear_regression function for calculating simple linear regression over the input time series on the selected time range. This function is useful for predictions and capacity planning. For example, range_linear_regression(process_resident_memory_bytes) can predict future memory usage based on the past memory usage.
FEATURE: MetricsQL: add range_stddev and range_stdvar functions.
FEATURE: MetricsQL: optimize expr1 op expr2 query when expr1 returns an empty result. In this case there is no sense in executing expr2 for op not equal to or, since the end result will be empty according to PromQL series matching rules. See this issue. Thanks to @jianglinjian for pointing to this case.
FEATURE: vmui: add the ability to upload/paste JSON to investigate the trace. See this issue and this pull request.
FEATURE: vmui: reduce JS bundle size from 200Kb to 100Kb. See this pull request.
FEATURE: vmui: add the ability to hide results of a particular query by clicking the eye icon. See this pull request.
FEATURE: vmui: add copy button to row on Table view. The button copies row in MetricQL format. See this issue.
FEATURE: vmui: add compact table view. See this issue.
FEATURE: vmui: add the ability to "stick" a tooltip on the chart by clicking on a data point. See this issue and this pull request
FEATURE: vmui: add the ability to set up series custom limits. See this issue.
FEATURE: vmalert: add default alert list for vmalert's metrics. See alerts-vmalert.yml.
FEATURE: vmagent: expose vmagent_relabel_config_*, vm_relabel_config_* and vm_promscrape_config_* metrics for tracking relabel and scrape configuration hot-reloads. See this issue.
BUGFIX: MetricsQL: properly return an empty result from limit_offset if the offset arg exceeds the number of inner time series. See this issue.
BUGFIX: vmagent: properly discover GCE zones when filter option is set at gce_sd_configs. See this issue.
BUGFIX: vmui: properly display the requested graph on the requested time range when navigating from Prometheus URL in Grafana.
BUGFIX: vmui: properly display wide tables. See this issue.
BUGFIX: reduce CPU usage spikes and memory usage spikes under high data ingestion rate introduced in v1.83.0. See this issue.

v1.83.1

Released at 2022-11-10

It is recommended upgrading to VictoriaMetrics v1.85.2 because of the bug, which may result in incomplete query results for historical time series.

FEATURE: vmagent: expose __meta_consul_partition label for targets discovered via consul_sd_configs in the same way as Prometheus 2.40 does.
FEATURE: vmui: show the query trace in JSON view. See this issue. Thanks to @michal-kralik for the pull request.
BUGFIX: VictoriaMetrics enterprise: fix a panic at vminsert when the discovered list of vmstorage nodes is changed during automatic vmstorage discovery. See this issue.
BUGFIX: properly register new time series in per-day inverted index if they were ingested during the last 10 seconds of the day. See this issue. Thanks to @lmarszal for the bugreport and for the initial fix.
BUGFIX: reduce the increased memory usage spikes for some workloads. The issue was introduced in v1.83.0.
BUGFIX: properly accept OpenTSDB telnet put lines without tags without the need to specify the trailing whitespace. See this issue.

v1.83.0

Released at 2022-10-29

It is recommended upgrading to VictoriaMetrics v1.85.2 because of the bug, which may result in incomplete query results for historical time series.

Update note 1: the indexdb/tagFilters cache type at /metrics has been renamed to indexdb/tagFiltersToMetricIDs in order to make its puropose more clear.

Update note 2: vmalert: the crlfEscape template function becames obsolete starting from this release. It can be safely removed from alerting templates, since \n chars are properly escaped with other *Escape functions now. See this and this issue for details.

FEATURE: VictoriaMetrics enterprise: add support for automatic vmstorage nodes discovering and updating at vmselect and vminsert. See these docs.
FEATURE: VictoriaMetrics enterprise: allow configuring multiple retentions for distinct sets of time series. See these docs, this and this feature request.
FEATURE: VictoriaMetric cluster enterprise: add support for multiple retentions for distinct tenants - see these docs and this and this feature request.
FEATURE: allow limiting memory usage on a per-query basis with -search.maxMemoryPerQuery command-line flag. See this feature request.
FEATURE: allow referring environment variables inside command-line flags via %{ENV_VAR} syntax. For example, if AUTH_KEY=top-secret environment variable is set, then -metricsAuthKey=%{AUTH_KEY} command-line flag is automatically expanded to -storageDataPath=top-secret at VictoriaMetrics startup. See these docs for details.
FEATURE: allow referring environment variables inside other environment variables via %{ENV_VAR} syntax. For example, if A=a-%{B}, B=b-%{C} and C=c env vars are set, then VictoriaMetrics components automatically expand them to A=a-b-c, B=b-c and C=c on startup.
FEATURE: vmagent: drop all the labels with __ prefix from discovered targets in the same way as Prometheus does according to this article. Previously the following labels were available during metric-level relabeling: __address__, __scheme__, __metrics_path__, __scrape_interval__, __scrape_timeout__, __param_*. Now these labels are available only during target-level relabeling. This should reduce CPU usage and memory usage for vmagent setups, which scrape big number of targets.
FEATURE: vmagent: improve the performance for metric-level relabeling, which can be applied via metric_relabel_configs section at scrape_configs, via -remoteWrite.relabelConfig or via -remoteWrite.urlRelabelConfig command-line options.

FEATURE: vmagent: allow specifying full url in scrape target addresses (aka __address__ label). This makes valid the following -promscrape.config:

scrape_configs:
- job_name: abc
  metrics_path: /foo/bar
  scheme: https
  static_configs:
  - targets:
    # the following targets are scraped by the provided full urls
    - 'http://host1/metric/path1'
    - 'https://host2/metric/path2'
    - 'http://host3:1234/metric/path3?arg1=value1'
    # the following target is scraped by <scheme>://host4:1234<metrics_path>
    - host4:1234

See the corresponding issue.

FEATURE: vmagent: allow controlling staleness tracking on a per-scrape_config basis by specifying no_stale_markers: true or no_stale_markers: false option in the corresponding scrape_config.
FEATURE: vmalert: add strvalue and stripDomain template functions in order to improve compatibility with Prometheus.
FEATURE: vmalert: add jsonEscape and htmlEscape template functions.
FEATURE: vmui: limit the number of plotted series. This should prevent from browser crashes or hangs when the query returns big number of time series. See this feature request.
FEATURE: vmui: reduce memory usage when querying big number of time series. See this issue.
FEATURE: vmui: add responsive styles for small screens. See this issue and this pull request.
FEATURE: log error if some environment variables referred at -promscrape.config via %{ENV_VAR} aren't found. This should prevent from silent using incorrect config files.
FEATURE: immediately shut down VictoriaMetrics apps on the second SIGINT or SIGTERM signal if they couldn't be finished gracefully for some reason after receiving the first signal.
FEATURE: improve the performance of /api/v1/series endpoint by eliminating loading of unused TSID data during the API call.
FEATURE: vmbackupmanager: add functionality for automated restore from backup. See these docs.
BUGFIX: MetricsQL: properly merge buckets with identical le values, but with different string representation of these values when calculating histogram_quantile and histogram_share. For example, http_request_duration_seconds_bucket{le="5"} and http_requests_duration_seconds_bucket{le="5.0"}. Such buckets may be returned from distinct targets. Thanks to @647-coder for the pull request.
BUGFIX: vmalert: change severity level for log messages about failed attempts for sending data to remote storage from error to warn. The message for about all failed send attempts remains at error severity level.
BUGFIX: vmalert: fix panic if vmalert runs with -clusterMode command-line flag in multitenant mode. The issue has been introduced in v1.82.0.
BUGFIX: vmalert: properly escape string passed to quotesEscape template function, so it can be safely embedded into JSON string. This makes obsolete the crlfEscape function. See this and this issue.
BUGFIX: vmagent: do not show invalid error message in Kubernetes service discovery: cannot parse WatchEvent json response: EOF. The invalid error message has been appeared in v1.82.0.
BUGFIX: vmagent: properly add exported_ prefix to metric labels, which clashing with scrape target labels if honor_labels: true option isn't set in scrape_config. Previously some exported_ prefixes were missing in the resulting metric labels. See this issue. The issue has been introduced in v1.82.0.
BUGFIX: vmselect: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} . This metric is used for monitoring rollup cache usage with the query vm_cache_size_bytes{type="promql/rollupResult"} / vm_cache_size_max_bytes{type="promql/rollupResult"} in the same way as this is done for other cache types.

v1.82.1

Released at 2022-10-14

BUGFIX: vmui: automatically update graph, legend and url after the removal of query field. See this feature request and this comment.
BUGFIX: vmalert: remove duplicate alertname JSON entry from generated alerts. See this issue. Thanks to @Howie59 for the fix!
BUGFIX: vmalert: fix integration with Grafana via -vmalert.proxyURL, which has been broken in v1.82.0. See this issue.
BUGFIX: vmbackup: set default region to us-east-1 if AWS_REGION environment variable isn't set. The issue was introduced in vmbackup v1.82.0. See this pull request.
BUGFIX: vmbackupmanager: fix deletion of old backups at Azure blob storage.
BUGFIX: MetricsQL: properly apply regex filters when searching for time series. Previously unexpected time series could be returned from regex filter. See this issue. The issue was introduced in v1.82.0.
BUGFIX: vmagent: properly apply if section with regex filters. Previously unexpected metrics could be returned from if section. The issue was introduced in v1.82.0.

v1.82.0

Released at 2022-10-07

It isn't recommended to use VictoriaMetrics and vmagent v1.82.0 because of the bug, which may result in incorrect query results and relabeling results. Upgrade to v1.82.1 instead.

Update note 1: this release changes data format for /api/v1/export/native in incompatible way, so it cannot be imported into older version of VictoriaMetrics via /api/v1/import/native.

Update note 2: vmalert changes default value for command-line flag -datasource.queryStep from 0s to 5m. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval.

Update note 3: vm_account_id and vm_project_id labels must be passed to tcp-based Graphite, InfluxDB and OpenTSDB endpoints at VictoriaMetrics cluster instead of undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels when writing samples to the needed tenant. See these docs for details.

FEATURE: VictoriaMetrics cluster: support specifying tenant ids via vm_account_id and vm_project_id labels. See these docs and this feature request.
FEATURE: vmagent: improve relabeling performance by up to 3x for non-trivial regex values such as ([^:]+):.+, which can be used for extracting a host part from host:port label value.
FEATURE: MetricsQL: improve performance by up to 4x for queries containing non-trivial regex filters such as {path=~"/foo/.+|/bar"}.
FEATURE: improve performance scalability on systems with many CPU cores for /federate and /api/v1/export/... endpoints.
FEATURE: sanitize metric names for data ingested via DataDog protocol according to DataDog metric naming. The behaviour can be disabled by passing -datadog.sanitizeMetricName=false command-line flag. Thanks to @PerGon for the pull request.
FEATURE: add -usePromCompatibleNaming command-line flag to vmagent, to single-node VictoriaMetrics and to vminsert component of VictoriaMetrics cluster. This flag can be used for normalizing the ingested metric names and label names to Prometheus-compatible form. If this flag is set, then all the chars unsupported by Prometheus are replaced with _ chars in metric names and labels of the ingested samples. See this feature request.
FEATURE: accept whitespace in metric names and tags ingested via Graphite plaintext protocol according to the specs. See this issue.
FEATURE: check the correctess of raw sample timestamps stored on disk when reading them. This reduces the probability of possible silent corruption of the data stored on disk. This should help this and this issue.
FEATURE: atomically delete directories with snapshots, parts and partitions at storage level. Previously such directories can be left in partially deleted state when the deletion operation was interrupted by unclean shutdown. This may result in cannot open file ...: no such file or directory error on the next start. The probability of this error was quite high when NFS or EFS was used as persistent storage for VictoriaMetrics data. See this issue.
FEATURE: set the start arg to end - 5 minutes if isn't passed explicitly to /api/v1/labels and /api/v1/label/.../values. See this pull request.
FEATURE: allow to define the minimum TLS version to use when accepting https requests to VictoriaMetrics components if -tls command-line flag is set. The minimum TLS version can be set via -tlsMinVersion command-line flag. See this feature request.
FEATURE: vmctl: add vm-native-step-interval command line flag for vm-native mode. New option allows splitting the import process into chunks by time interval. This helps migrating data sets with high churn rate and provides better control over the process. See feature request.
FEATURE: vmui: add top queries tab, which shows various stats for recently executed queries. See these docs and this feature request.
FEATURE: vmui: move the "Execute Query" and "Add Query" buttons below the query fields, change icon for remove query. See this issue.
FEATURE: vmui: set the maximum number of queries to 4, remove multi Y-axes, left one for all queries and dotted lines to indicate queries in the graph. See this issue.
FEATURE: vmalert: add debug mode to the alerting rule settings for printing additional information into logs during evaluation. See debug param in alerting rule config.
FEATURE: vmalert: add experimental feature for displaying last 10 states of the rule (recording or alerting) evaluation. The state is available on the Rule page, which can be opened by clicking on Details link next to Rule's name on the /groups page.
FEATURE: vmalert: allow using extra labels in annotiations. See this feature request.
FEATURE: vmalert: allow configuring authorization params per list of targets in vmalert's notifier config for static_configs. See this issue.
FEATURE: vmalert: allow using {% raw %}{{$labels}}{% endraw %} for templating in command-line flag -external.alert.source. The change supposed to provide additional flexibility for generating alert's source link based on labels values.
FEATURE: vmalert: add vm_account_id and vm_project_id labels to results of alerting and recording rules if -clusterMode is enabled. This improves multitenant support in vmalert.
FEATURE: vmagent: minimize the time needed for reading large responses from scrape targets in stream parsing mode. This should reduce scrape durations for such targets as kube-state-metrics running in a big Kubernetes cluster.
FEATURE: MetricsQL: add sort_by_label_numeric and sort_by_label_numeric_desc functions for numeric sort of input time series by the specified labels. See this feature request.
FEATURE: vmbackup and vmrestore: retry GCS operations for up to 3 minutes on temporary failures. See this issue.
FEATURE: vmbackup: add support for saving / restoring backups to / from Azure blob storage. See this feature request.
FEATURE: vmbackupmanager: expose vm_backup_in_flight metric, which can be used for determining which backup types - latest, hourly, daily, weekly or monthly - are currently executed.
FEATURE: vmgateway: add ability to extract JWT authorization token from non-standard HTTP header by passing it via -auth.httpHeader command-line flag. See this feature request.
FEATURE: vmagent: expose __meta_ec2_region label for ec2_sd_config in the same way as Prometheus 2.39 does.
FEATURE: vmagent: accept data ingestion requests via paths starting from /prometheus prefix in the same way as VictoriaMetrics does. For example, vmagent now accepts Prometheus remote_write data via both /api/v1/write and /prometheus/api/v1/write. This simplifies switching between single-node VictoriaMetrics and vmagent.
FEATURE: vmagent: add external_labels from global section at -promscrape.config after the relabeling is applied to scraped metrics. This aligns with Prometheus behaviour. Previously the external_labels were added to scrape targets, so they could be modified during relabeling. See this issue.
FEATURE: vmagent: allow specifying per--remoteWrite.url limits for on-disk size for pending data via -remoteWrite.maxDiskUsagePerURL command-line flag. Thanks to @rbizos for the pull request.
FEATURE: VictoriaMetrics cluster: log clear error when multiple identical -storageNode command-line flags are passed to vmselect or to vminsert. Previously these components were crashed with cryptic panic metric ... is already registered in this case. See this issue.
BUGFIX: do not export stale metrics via /federate api after the staleness markers. Previously such metrics were exported with NaN values. this could break some setups. See this issue.
BUGFIX: export ininity numbers as "Infinity" strings at /api/v1/export, so they can be parsed by standard JSON parsers. Previously infinity numbers were exported as Inf values, which couldn't be parsed by standard JSON parsers. See this issue.
BUGFIX: vmauth: properly handle request paths ending with / such as /vmui/. Previously vmui was dropping the traling /, which could prevent from using vmui via vmauth. See this issue.
BUGFIX: vmagent: properly encode query params for aws signed requests, use %20 instead of + as api requires. See this issue.
BUGFIX: vmagent: properly parse relabel config when regex ending with escaped $. See this issue.
BUGFIX: MetricsQL: properly calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d. Previously the sum_over_time(m[d]) could be improperly divided by smaller than d time range. See rate_over_sum() docs and this issue.
BUGFIX: MetricsQL: properly calculate increase(m[d]) over slow-changing counters with values smaller than 100. Previously increase could return unexpectedly big results in this case. See the related issue and this pull request.
BUGFIX: MetricsQL: ignore empty series when applying limit_offset. It should improve queries with additional filters by value in expressions like limit_offset(1,1, foo > 1).
BUGFIX: MetricsQL: properly calculate quantiles_over_time when the lookbehind window contains only a single sample. Previously an empty result was incorrectly returned in this case.
BUGFIX: vmui: fix RangeError: Maximum call stack size exceeded error when the query returns too many data points at Table view. See this pull request.
BUGFIX: vmui: fix workaround for adding more queries via URL. See this issue.
BUGFIX: vmalert: re-evaluate annotations per each alert evaluation. Previously, annotations were evaluated only on alert's value change. This could result in stale annotations in some cases described in this pull request.
BUGFIX: prevent from excessive CPU usage when the storage enters read-only mode. The previous fix in v1.81.0 wasn't complete.
BUGFIX: vmalert: change default value for command-line flag -datasource.queryStep from 0s to 5m. Param step is added by vmalert to every rule evaluation request sent to datasource. Before this change, step was equal to group's evaluation interval by default. Param step for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval.
BUGFIX: properly calculate vm_rows_scanned_per_query histogram exported at /metrics page of vmselect and single-node VictoriaMetrics. Previously it could return misleadingly high numbers for rollup functions, which scan only a few samples on the provided lookbehind window in square brackets. For example, increase(m[1d]) always scans only 2 rows (aka raw samples) per each returned time series.

v1.81.2

Released at 2022-09-08

BUGFIX: VictoriaMetrics cluster: properly calculate query results at vmselect. See this issue. The issue has been introduced in v1.81.0.

v1.81.1

Released at 2022-09-02

It isn't recommended to use VictoriaMetrics cluster v1.81.1 because of the bug, which may result in incorrect query results. Upgrade to v1.81.2 instead.

FEATURE: MetricsQL: evaluate q1, ..., qN in parallel when calculating union(q1, .., qN). Previously union args were evaluated sequentially. This could result in lower than expected performance.
BUGFIX: VictoriaMetrics cluster: fix potential panic at vmselect under high load, which has been introduced in v1.81.0. See this issue.

v1.81.0

It isn't recommended to use VictoriaMetrics cluster v1.81.0 because of the bug, which may result in vmselect crashes under high load. Upgrade to v1.81.2 instead.

Released at 2022-08-31

Update note 1: vmalert by default hides values of -remoteWrite.url, -remoteRead.url and -datasource.url in logs and at http://vmalert:8880/flags for security reasons. See the corresponding SECURITY change in the Chagelog below for additional info.

Update note 2: vmalert by default points alert source url to /vmalert/alert?... aka web UI instead of /vmalert/api/v1/alert?... aka JSON handler. The old behavior can be achieved by setting {% raw %}-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %} command-line flag.

SECURITY: vmalert: do not expose -remoteWrite.url, -remoteRead.url and -datasource.url command-line flag values in logs and at http://vmalert:8880/flags page by default, since they may contain sensitive data such as auth keys. This aligns vmalert behaviour with vmagent, which doesn't expose -remoteWrite.url command-line flag value in logs and at http://vmagent:8429/flags page by default. Specify -remoteWrite.showURL, -remoteRead.showURL and -datasource.showURL command-line flags for showing values for the corresponding -*.url flags in logs. Thanks to @mble for the pull request.
SECURITY: upgrade base docker image (alpine) from 3.16.1 to 3.16.2. See alpine 3.16.2 release notes.
FEATURE: return shorter error messages to Grafana and to other clients requesting /api/v1/query and /api/v1/query_range endpoints. This should simplify reading these errors by humans. The long error message with full context is still written to logs.
FEATURE: add the ability to fine-tune the number of points, which can be generated per each matching time series during subquery evaluation. This can be done with the -search.maxPointsSubqueryPerTimeseries command-line flag. See this feature request.
FEATURE: vmagent: improve the performance for relabeling rules with commonly used regular expressions in regex and if fields such as some_string, prefix.*, prefix.+, foo|bar|baz, .*foo.* and .+foo.+.
FEATURE: vmagent: reduce CPU usage when discovering big number of Kubernetes targets with big number of labels and annotations.
FEATURE: vmagent: add ability to accept multitenant data via OpenTSDB /api/put protocol at /insert/<tenantID>/opentsdb/api/put http endpoint if multitenant support is enabled at vmagent. Thanks to @chengjianyun for the pull request.
FEATURE: monitoring: expose vm_hourly_series_limit_max_series, vm_hourly_series_limit_current_series, vm_daily_series_limit_max_series and vm_daily_series_limit_current_series metrics when -search.maxHourlySeries or -search.maxDailySeries limits are set. This allows alerting when the number of unique series reaches the configured limits. See these docs for details.
FEATURE: VictoriaMetrics cluster: reduce the amounts of logging at vmstorage when vmselect connects/disconnects to vmstorage.
FEATURE: VictoriaMetrics cluster: improve performance for heavy queries on systems with many CPU cores.
FEATURE: vmagent: add ability to use {% raw %}{{label_name}}{% endraw %} placeholders in the replacement option of relabeling rules. This simplifies constructing label values from multiple existing label values. See these docs for details.
FEATURE: vmagent: generate additional per-target metrics - scrape_series_limit, scrape_series_current and scrape_series_limit_samples_dropped if series limit is set according to these docs. This simplifies alerting on targets with the exceeded series limit. See these docs for details on these metrics.
FEATURE: vmagent: add support for MX record types in dns_sd_configs in the same way as Prometheus 2.38 does.
FEATURE: vmagent: add __meta_kubernetes_service_port_number meta-label for role: service in kubernetes_sd_configs in the same way as Prometheus 2.38 does.
FEATURE: vmagent: add __meta_kubernetes_pod_container_image meta-label for role: pod in kubernetes_sd_configs in the same way as Prometheus 2.38 does.
FEATURE: vmagent: retry HTTP requests after some wait time during service discovery and during target scrapes if the server returns 429 HTTP status code (aka Too many requests). See this issue.
FEATURE: vmui: add a legend in the top right corner for shortcut keys. See this feature request.
FEATURE: vmalert: add toTime() template function in the same way as Prometheus 2.38 does. See these docs.
FEATURE: vmalert: add $alertID and $groupID template variables. These variables may be used for templating annotations or -external.alert.source command-line flag. See the full list of supported variables here.
FEATURE: vmalert: add $activeAt template variable. See this feature request. See the full list of supported variables here. Thanks to @laixintao for the pull request.
FEATURE: vmalert: point alert source to vmalert's UI at /vmalert/alert?... instead of JSON handler at /vmalert/api/v1/alert?.... This improves user experience. The old behavior can be achieved by setting {% raw %}-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %} command-line flag.
BUGFIX: prevent from excess CPU usage when the storage enters read-only mode.
BUGFIX: improve performance for requests to /api/v1/labels and /api/v1/label/.../values when the filter in the match[] query arg matches small number of time series. The performance for this case has been reduced in v1.78.0. See this and this issues.
BUGFIX: increase the default limit on the number of concurrent merges for small parts from 8 to 16. This should help resolving potential issues with heavy data ingestion. See this comment from @lukepalmer .
BUGFIX: MetricsQL: fix panic when incorrect arg is passed as phi into histogram_quantiles function. See this issue.

v1.80.0

Released at 2022-08-08

FEATURE: vmalert: allow configuring additional HTTP request headers for -datasource.url, -remoteWrite.url and -remoteRead.url via -datasource.headers, -remoteWrite.headers and -remoteRead.headers command-line flags. Additional HTTP request headers also can be set on group level via headers param - see these docs and this issue.
FEATURE: MetricsQL: execute left and right sides of certain operations in parallel. For example, q1 or q2, aggr_func(q1) <op> q2, q1 <op> aggr_func(q1). This may improve query performance if VictoriaMetrics has enough free resources for parallel processing of both sides of the operation. See this feature request.
FEATURE: vmauth: allow multiple sections with duplicate username but with different password values at -auth.config file.
FEATURE: add ability to push internal metrics (e.g. metrics exposed at /metrics page) to the configured remote storage from all the VictoriaMetrics components. See these docs.
FEATURE: improve performance for heavy queries over big number of time series on systems with big number of CPU cores. See this issue. Thanks to @zqyzyq for the idea.
FEATURE: improve performance for registering new time series in indexdb by up to 50%. Thanks to @ahfuzhang for the issue.
FEATURE: vmagent: add ability to specify tenantID in target labels. In this case metrics from the given target are routed to the given __tenant_id__. See these docs and this feature request.
FEATURE: vmagent: add service discovery for Yandex Cloud. See these docs and this feature request.
FEATURE: vmui. Zoom in the graph by selecting the needed time range in the same way Grafana does. Hold ctrl (or cmd on MacOS) in order to move the graph to the left/right. Hold ctrl (or cmd on MacOS) and scroll up/down in order to zoom in/out the area under the cursor. See this feature request.
BUGFIX: VictoriaMetrics cluster: fix potential panic in multi-level cluster setup when top-level vmselect is configured with -replicationFactor bigger than 1. See this issue.
BUGFIX: vmagent: properly handle custom endpoint value in ec2_sd_configs. It was ignored since v1.77.0 because of a bug in the implementation of this feature request. See this issue.
BUGFIX: vmagent: add missing __meta_kubernetes_ingress_class_name meta-label for role: ingress service discovery in Kubernetes. See this commit from Prometheus.
BUGFIX: vmagent: allow stale responses from Consul service discovery (aka consul_sd_configs) by default in the same way as Prometheus does. This should reduce load on Consul when discovering big number of targets. Stale responses can be disabled by specifying allow_stale: false option in consul_sd_config. See this issue.
BUGFIX: vmagent: dockerswarm_sd_configs: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels as Prometheus does. See this issue.
BUGFIX: vmagent: set up metric to 0 for partial scrapes in stream parsing mode. Previously the up metric was set to 1 when at least a single metric has been scraped before the error. This aligns the behaviour of vmselect with Prometheus.
BUGFIX: vmagent: restart all the scrape jobs during config reload after global section is changed inside -promscrape.config. See this issue.
BUGFIX: vmagent: properly assume role with AWS ECS credentials. See this issue. Thanks to @transacid for the fix.
BUGFIX: vmagent: do not split regex in relabeling rules into multiple lines if it contains groups. This fixes this issue.
BUGFIX: MetricsQL: return series from q1 if q2 doesn't return matching time series in the query q1 ifnot q2. Previously series from q1 weren't returned in this case.
BUGFIX: vmui: properly show date picker at Table tab. See this issue.
BUGFIX: properly generate http redirects if -http.pathPrefix command-line flag is set. See this issue.