From 8cb1822b9404e0ab5cb203dd2f22de32fc0099c0 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Thu, 25 Jul 2024 12:47:35 +0200 Subject: [PATCH] Revert "removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)" This reverts commit 2e9b1efeb9422c60a345c67ebbbcaa5f8d37469d. Reason for revert: relative links in docs are much harder to maintain in consistent state comparing to absolute links: - It is non-trivial to figure out the proper relative link path when creating and editing docs. - Relative links break after moving the doc files to another paths, and it is non-trivial to figure which links are broken after that. - The updated relative links do not work properly right now in the docs. For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page. This is documented at https://docs.victoriametrics.com/#images-in-documentation . --- docs/CHANGELOG.md | 1022 +++++++++++++-------------- docs/CHANGELOG_2020.md | 44 +- docs/CHANGELOG_2021.md | 278 ++++---- docs/CHANGELOG_2022.md | 976 +++++++++++++------------- docs/CHANGELOG_2023.md | 1156 +++++++++++++++---------------- docs/Cluster-VictoriaMetrics.md | 370 +++++----- docs/VictoriaLogs/CHANGELOG.md | 292 ++++---- docs/VictoriaLogs/LogsQL.md | 558 +++++++-------- docs/data-ingestion/README.md | 8 +- docs/data-ingestion/Vector.md | 2 +- 10 files changed, 2353 insertions(+), 2353 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index fb9c7f292..23af4fe2b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -11,25 +11,25 @@ aliases: --- The following `tip` changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs: -* [How to build single-node VictoriaMetrics](./README.md#how-to-build-from-sources) -* [How to build cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#building-from-sources) -* [How to build vmagent](./vmagent.md#how-to-build-from-sources) -* [How to build vmalert](./vmalert.md#how-to-build-from-sources) -* [How to build vmauth](./vmauth.md#how-to-build-from-sources) -* [How to build vmbackup](./vmbackup.md#how-to-build-from-sources) -* [How to build vmrestore](./vmrestore.md#how-to-build-from-sources) -* [How to build vmctl](./vmctl.md#how-to-build) +* [How to build single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-build-from-sources) +* [How to build cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#building-from-sources) +* [How to build vmagent](https://docs.victoriametrics.com/vmagent/#how-to-build-from-sources) +* [How to build vmalert](https://docs.victoriametrics.com/vmalert/#how-to-build-from-sources) +* [How to build vmauth](https://docs.victoriametrics.com/vmauth/#how-to-build-from-sources) +* [How to build vmbackup](https://docs.victoriametrics.com/vmbackup/#how-to-build-from-sources) +* [How to build vmrestore](https://docs.victoriametrics.com/vmrestore/#how-to-build-from-sources) +* [How to build vmctl](https://docs.victoriametrics.com/vmctl/#how-to-build) Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our [sandbox](https://play-grafana.victoriametrics.com/d/oS7Bi_0Wz_vm/victoriametrics-cluster-vm). The sandbox cluster installation runs under the constant load generated by [prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) and is used for testing latest releases. -See also [LTS releases](./LTS-releases.md). +See also [LTS releases](https://docs.victoriametrics.com/lts-releases/). ## tip -**Update note 1: [vmauth](./vmauth.md) HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align [vmauth](./vmauth.md) behaviour with other well-known reverse-proxies behaviour. ** +**Update note 1: [vmauth](https://docs.victoriametrics.com/vmauth/) HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align [vmauth](https://docs.victoriametrics.com/vmauth/) behaviour with other well-known reverse-proxies behaviour. ** * SECURITY: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2. See [alpine 3.20.2 release notes](https://alpinelinux.org/posts/Alpine-3.20.2-released.html). @@ -37,49 +37,49 @@ See also [LTS releases](./LTS-releases.md). * FEATURE: [vmauth](./vmauth.md): add `keep_original_host` option, which can be used for proxying the original `Host` header from client request to the backend. By default the backend host is used as `Host` header when proxying requests to the configured backends. See [these docs](./vmauth.md#host-http-header). * FEATURE: [vmauth](./vmauth.md) now returns HTTP 502 status code when all upstream backends are not available. Previously, it returned HTTP 503 status code. This change aligns vmauth behavior with other well-known reverse-proxies behavior. -* BUGFIX: [vmauth](./vmauth.md): properly proxy requests to backend urls ending with `/` if the original request path equals to `/`. Previously the trailing `/` at the backend path was incorrectly removed. For example, if the request to `http://vmauth/` is configured to be proxied to `url_prefix=http://backend/foo/`, then it was proxied to `http://backend/foo`, while it should go to `http://backend/foo/`. -* BUGFIX: [vmauth](./vmauth.md): fix `cannot read data after closing the reader` error when proxying HTTP requests without body (aka `GET` requests). The issue has been introduced in [v1.102.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0) in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7ee57974935a662896f2de40fdf613156630617d). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly proxy requests to backend urls ending with `/` if the original request path equals to `/`. Previously the trailing `/` at the backend path was incorrectly removed. For example, if the request to `http://vmauth/` is configured to be proxied to `url_prefix=http://backend/foo/`, then it was proxied to `http://backend/foo`, while it should go to `http://backend/foo/`. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix `cannot read data after closing the reader` error when proxying HTTP requests without body (aka `GET` requests). The issue has been introduced in [v1.102.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0) in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7ee57974935a662896f2de40fdf613156630617d). ## [v1.102.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0) Released at 2024-07-17 -**Update note 1: support for snap packages was removed due to lack of interest from community. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543) for details. Please read about supported package types [here](./README.md#install).** +**Update note 1: support for snap packages was removed due to lack of interest from community. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543) for details. Please read about supported package types [here](https://docs.victoriametrics.com/#install).** -**Update note 2: [stream aggregation config](./stream-aggregation.md#stream-aggregation-config) now prevents setting multiple identical outputs. For example, `outputs: [total, total]` will fail the validation phase. In addition, `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` will fail the validation as well - use `outputs: ["quantiles(0.5, 0.9)"]` instead.** +**Update note 2: [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) now prevents setting multiple identical outputs. For example, `outputs: [total, total]` will fail the validation phase. In addition, `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` will fail the validation as well - use `outputs: ["quantiles(0.5, 0.9)"]` instead.** * SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in [Go1.22.5](https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See [alpine 3.20.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html). -* FEATURE: [streaming aggregation](./stream-aggregation.md): expose the following metrics at `/metrics` page of [vmagent](./vmagent.md) and [single-node VictoriaMetrics](./README.md): +* FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): expose the following metrics at `/metrics` page of [vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/): * `vm_streamaggr_matched_samples_total` - the number of input samples matched by the corresponding aggregation rule * `vm_streamaggr_output_samples_total` - the number of output samples produced by the corresponding aggregation rule - * `vm_streamaggr_samples_lag_seconds` - [histogram](./keyConcepts.md#histogram) with the lag between the current time and the timestamp seen in the aggregated input samples -* FEATURE: [steaming aggregation](./stream-aggregation.md): add new labels to `vm_streamaggr_*` metrics: - * `name` - the name of the streaming aggregation rule, which can be configured via `name` option - see [these docs](./stream-aggregation.md#stream-aggregation-config). + * `vm_streamaggr_samples_lag_seconds` - [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) with the lag between the current time and the timestamp seen in the aggregated input samples +* FEATURE: [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): add new labels to `vm_streamaggr_*` metrics: + * `name` - the name of the streaming aggregation rule, which can be configured via `name` option - see [these docs](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). * `url` - `-remoteWrite.url` for the corresponding `-remoteWrite.streamAggr.config` * `path` - path to the corresponding streaming aggregation config file * `position` - the position of the aggregation rule in the corresponding streaming aggregation config file -* FEATURE: [streaming aggregation](./stream-aggregation.md): prevent having duplicated aggregation function as `outputs` in one [aggregation config](./stream-aggregation.md#stream-aggregation-config). It also prevents using `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` instead of `outputs: ["quantiles(0.5, 0.9)"]`, as the former has higher computation cost for producing the same result. -* FEATURE: [vmagent](./vmagent.md) and [Single-node VictoriaMetrics](./README.md): add `-graphite.sanitizeMetricName` command-line flag for sanitizing metrics ingested via [Graphite protocol](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077). -* FEATURE: [vmagent](./vmagent.md): [`yandexcloud_sd_configs`](./sd_configs.md#yandexcloud_sd_configs): add support for obtaining IAM token in [GCE format](https://yandex.cloud/en-ru/docs/compute/operations/vm-connect/auth-inside-vm#auth-inside-vm) additionally to the [deprecated Amazon EC2 IMDSv1 format](https://yandex.cloud/en/docs/security/standard/authentication#aws-token). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513). -* FEATURE: [vmalert](./vmalert.md): make `-replay.timeTo` optional in [replay mode](./vmalert.md#rules-backfilling). When omitted, the current timestamp will be used. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6492). -* FEATURE: [vmauth](./vmauth.md): reduce CPU usage when proxying data ingestion requests. -* FEATURE: [vmauth](./vmauth.md): allow disabling request body caching with `-maxRequestBodySizeToRetry=0`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445). Thanks to @shichanglin5 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533). -* FEATURE: [vmauth](./vmauth.md): allow overriding `Host` header with backend host before sending the request to the configured backend. See [these docs](./vmauth.md#modifying-http-headers) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6453). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): do not retry RPC calls to vmstorage nodes if [complexity limits](./README.md#resource-usage-limits) were exceeded. -* FEATURE: [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md): add support for [Azure Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) and default credentials lookup. See [these docs](./vmbackup.md#providing-credentials-via-env-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984) for the details. Thanks to @justinrush for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518). -* FEATURE: [vmbackup](./vmbackup.md) and [vmrestore](./vmbackup.md): allow overriding Azure storage domain when performing backups via `AZURE_STORAGE_DOMAIN` environment variable. See [these docs](./vmbackup.md#providing-credentials-via-env-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984). Thanks to @justinrush for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518). -* FEATURE: [vmui](./README.md#vmui): show compacted result in the JSON tab for query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6559). -* FEATURE: [dashboards](https://grafana.com/orgs/victoriametrics): add [Grafana dashboard](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/vmauth.json) and [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml) for [vmauth](./vmauth.md) dashboard. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4313) for details. +* FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent having duplicated aggregation function as `outputs` in one [aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). It also prevents using `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` instead of `outputs: ["quantiles(0.5, 0.9)"]`, as the former has higher computation cost for producing the same result. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/) and [Single-node VictoriaMetrics](https://docs.victoriametrics.com/): add `-graphite.sanitizeMetricName` command-line flag for sanitizing metrics ingested via [Graphite protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): [`yandexcloud_sd_configs`](https://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs): add support for obtaining IAM token in [GCE format](https://yandex.cloud/en-ru/docs/compute/operations/vm-connect/auth-inside-vm#auth-inside-vm) additionally to the [deprecated Amazon EC2 IMDSv1 format](https://yandex.cloud/en/docs/security/standard/authentication#aws-token). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): make `-replay.timeTo` optional in [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling). When omitted, the current timestamp will be used. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6492). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): reduce CPU usage when proxying data ingestion requests. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow disabling request body caching with `-maxRequestBodySizeToRetry=0`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445). Thanks to @shichanglin5 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow overriding `Host` header with backend host before sending the request to the configured backend. See [these docs](https://docs.victoriametrics.com/vmauth/#modifying-http-headers) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6453). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): do not retry RPC calls to vmstorage nodes if [complexity limits](https://docs.victoriametrics.com/#resource-usage-limits) were exceeded. +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/): add support for [Azure Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) and default credentials lookup. See [these docs](https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984) for the details. Thanks to @justinrush for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmbackup/): allow overriding Azure storage domain when performing backups via `AZURE_STORAGE_DOMAIN` environment variable. See [these docs](https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984). Thanks to @justinrush for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show compacted result in the JSON tab for query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6559). +* FEATURE: [dashboards](https://grafana.com/orgs/victoriametrics): add [Grafana dashboard](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/vmauth.json) and [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml) for [vmauth](https://docs.victoriametrics.com/vmauth/) dashboard. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4313) for details. * FEATURE: [vmagent dashboard](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/vmagent.json): `stream aggregation` section: add graphs based on newly exposed streaming aggregation metrics. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [histogram_quantile](./MetricsQL.md#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). -* BUGFIX: [vmagent](./vmagent.md): fix panic when using multiple topics with the same name when [ingesting metrics from Kafka](./vmagent.md#kafka-integration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636) for the details. -* BUGFIX: [vmalert](./vmalert.md): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). -* BUGFIX: [vmgateway](./vmgateway.md): properly apply read and write based rate limits. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6148) for details. -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix `vm_backup_last_run_failed` metric not being properly initialized during startup. Previously, it could imply an error even if the backup have been completed successfully. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6550) for the details. -* BUGFIX: [vmui](./README.md#vmui): fix input cursor position reset in modal settings. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6530). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix panic when using multiple topics with the same name when [ingesting metrics from Kafka](https://docs.victoriametrics.com/vmagent/#kafka-integration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636) for the details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). +* BUGFIX: [vmgateway](https://docs.victoriametrics.com/vmgateway/): properly apply read and write based rate limits. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6148) for details. +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix `vm_backup_last_run_failed` metric not being properly initialized during startup. Previously, it could imply an error even if the backup have been completed successfully. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6550) for the details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix input cursor position reset in modal settings. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6530). * BUGFIX: [docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#docker-compose-environment-for-victoriametrics): fix incorrect link to vmui from [VictoriaMetrics plugin in Grafana](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#grafana). ## [v1.102.0-rc2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0-rc2) @@ -91,35 +91,35 @@ Released at 2024-06-24 **Update note 2: `*.passwordFile` and similar flags are no longer trimming trailing whitespaces at the end of content. Make sure to update the templating of password files or HTTP endpoints to not include trailing whitespaces before the upgrade. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6503) PR for the details.** * FEATURE: [alerts-vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmagent.yml): add new alerting rules `StreamAggrFlushTimeout` and `StreamAggrDedupFlushTimeout` to notify about issues during stream aggregation. -* FEATURE: [dashboards/vmagent](https://grafana.com/grafana/dashboards/12683): add row `Streaming aggregation` with panels related to [streaming aggregation](./stream-aggregation.md) process. -* FEATURE: [vmauth](./vmauth.md): add `-idleConnTimeout` flag set to 50s by default. It should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmauth logs. -* FEATURE: [vmauth](./vmauth.md): add automatic retry for requests to backend for trivial network errors, such as `broken pipe` and `connection reset` for requests to the configured backends. -* FEATURE: [vmagent](./vmagent.md): increase default value of `-promscrape.maxDroppedTargets` command-line flag to 10_000 from 1000. This makes it easier to track down large number of dropped targets. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381). -* FEATURE: [vmagent](./vmagent.md): add `max_scrape_size` option to [scrape config](./sd_configs.md#scrape_configs) for setting custom limit on the response size target can send. The new [automatically generated metric](./vmagent.md#automatically-generated-metrics) `scrape_response_size_bytes` is added to reflect the response size of the target. See these issues: [1](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429), [2](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2992), [3](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6123), [4](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5612). -* FEATURE: [vmsingle](./Single-Server-VictoriaMetrics.md): check for ranged vector arguments in non-rollup expressions when `-search.disableImplicitConversion` or `-search.logImplicitConversion` are enabled. For example, `sum(up[5m])` or `absent(up[5m])` will fail to execute if these flags are set. -* FEATURE: [vmsingle](./Single-Server-VictoriaMetrics.md): validate that rollup expressions has ranged vector arguments passed when `-search.disableImplicitConversion` or `-search.logImplicitConversion` are enabled. For example, `rate(metric)` or `count_over_time(metric)` will fail to execute if these flags are set. -* FEATURE: [vmalert-tool](./vmalert-tool.md): support file path with hierarchical patterns and regexpes, and http url in unittest cmd-line flag `-files`, e.g. `-files="http:///path/to/rules"` or `-files="dir/**/*.yaml"`. +* FEATURE: [dashboards/vmagent](https://grafana.com/grafana/dashboards/12683): add row `Streaming aggregation` with panels related to [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/) process. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add `-idleConnTimeout` flag set to 50s by default. It should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmauth logs. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add automatic retry for requests to backend for trivial network errors, such as `broken pipe` and `connection reset` for requests to the configured backends. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): increase default value of `-promscrape.maxDroppedTargets` command-line flag to 10_000 from 1000. This makes it easier to track down large number of dropped targets. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `max_scrape_size` option to [scrape config](https://docs.victoriametrics.com/sd_configs/#scrape_configs) for setting custom limit on the response size target can send. The new [automatically generated metric](https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics) `scrape_response_size_bytes` is added to reflect the response size of the target. See these issues: [1](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429), [2](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2992), [3](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6123), [4](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5612). +* FEATURE: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/): check for ranged vector arguments in non-rollup expressions when `-search.disableImplicitConversion` or `-search.logImplicitConversion` are enabled. For example, `sum(up[5m])` or `absent(up[5m])` will fail to execute if these flags are set. +* FEATURE: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/): validate that rollup expressions has ranged vector arguments passed when `-search.disableImplicitConversion` or `-search.logImplicitConversion` are enabled. For example, `rate(metric)` or `count_over_time(metric)` will fail to execute if these flags are set. +* FEATURE: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/): support file path with hierarchical patterns and regexpes, and http url in unittest cmd-line flag `-files`, e.g. `-files="http:///path/to/rules"` or `-files="dir/**/*.yaml"`. * BUGFIX: all VictoriaMetrics components: properly calculate `process_resident_memory_bytes` metric for OS with non-default value of memory page size. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457) for details. * BUGFIX: all VictoriaMetrics components: prioritize `-configAuthKey` and `-reloadAuthKey` over `-httpAuth.*` settings. This change aligns behavior of mentioned flags with other auth flags like `-metricsAuthKey`, `-flagsAuthKey`, `-pprofAuthKey`. Check [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329). * BUGFIX: all VictoriaMetrics components: do not trim trailing spaces when reading content from `*.passwordFile` and similar flags. Previously, trailing spaces were trimmed from the content of the password file, which could lead to unexpected authentication errors. -* BUGFIX: [vmctl](./vmctl.md): add `--disable-progress-bar` global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. `--vm-disable-progress-bar` command-line flag is deprecated and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367). -* BUGFIX: [stream aggregation](./stream-aggregation.md): prevent [rate_sum](./stream-aggregation.md#rate_sum) and [rate_avg](./stream-aggregation.md#rate_avg) producing `NaN` results for stale time series. Before, when series matched for aggregation became stale or weren't updated during aggregation interval, the `rate_sum` or `rate_avg` could produce data point with `NaN` value. During visualization, such aggregation results would be displayed as gaps in time series. -* BUGFIX: [vmalert](./vmalert.md): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433). -* BUGFIX: [vmalert](./vmalert.md): exit replay mode with non-zero code if generated samples are not successfully written into remoteWrite url. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512). -* BUGFIX: [vmbackup](./vmbackup.md): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353). -* BUGFIX: [vmalert](./vmalert.md) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. -* BUGFIX: [vmalert](./vmalert.md): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add validation for the max value specified for `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330) for details. -* BUGFIX: [vmui](./README.md#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). -* BUGFIX: [vmalert-tool](./vmalert-tool.md): prevent hanging when processing groups without rules. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500). -* BUGFIX: [vmauth](./vmauth.md): fix discovering backend IPs when `url_prefix` contains hostname with srv+ prefix. Thanks to @shichanglin5 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--disable-progress-bar` global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. `--vm-disable-progress-bar` command-line flag is deprecated and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent [rate_sum](https://docs.victoriametrics.com/stream-aggregation/#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/#rate_avg) producing `NaN` results for stale time series. Before, when series matched for aggregation became stale or weren't updated during aggregation interval, the `rate_sum` or `rate_avg` could produce data point with `NaN` value. During visualization, such aggregation results would be displayed as gaps in time series. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): exit replay mode with non-zero code if generated samples are not successfully written into remoteWrite url. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add validation for the max value specified for `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330) for details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). +* BUGFIX: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/): prevent hanging when processing groups without rules. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix discovering backend IPs when `url_prefix` contains hostname with srv+ prefix. Thanks to @shichanglin5 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401). ## [v1.102.0-rc1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0-rc1) Released at 2024-06-07 -**Update note 1: the `-remoteWrite.multitenantURL` command-line flag at `vmagent` was removed starting from this release. This flag was deprecated since [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0). Use `-enableMultitenantHandlers` instead, as it is easier to use and combine with [multitenant URL at vminsert](./Cluster-VictoriaMetrics.md#multitenancy-via-labels). See [these docs](./vmagent.md#multitenancy) for details.** +**Update note 1: the `-remoteWrite.multitenantURL` command-line flag at `vmagent` was removed starting from this release. This flag was deprecated since [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0). Use `-enableMultitenantHandlers` instead, as it is easier to use and combine with [multitenant URL at vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels). See [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy) for details.** **Update note 2: the `-streamAggr.dropInputLabels` command-line flag at `vmagent` was renamed to `-remoteWrite.streamAggr.dropInputLabels`. `-streamAggr.dropInputLabels` is now used for global streaming aggregation.** @@ -135,175 +135,175 @@ Released at 2024-06-07 * FEATURE: [dashboards/operator](https://grafana.com/grafana/dashboards/17869), [dashboards/backupmanager](https://grafana.com/grafana/dashboards/17798) and [dashboard/tenant-statistic](https://grafana.com/grafana/dashboards/16399): update dashboard to be compatible with Grafana 10+ version. * FEATURE: [dashboards/cluster](https://grafana.com/grafana/dashboards/11176): add new panel `Concurrent selects` to `vmstorage` row. The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details. * FEATURE: [dashboards](https://grafana.com/orgs/victoriametrics): use `$__interval` variable for offsets and look-behind windows in annotations. This should improve precision of `restarts` and `version change` annotations when zooming-in/zooming-out on the dashboards. -* FEATURE: [vmagent](./vmagent.md): support aggregation and deduplication configs before replicating data to configured `-remoteWrite.url` destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467). -* FEATURE: [vmagent](./vmagent.md): add service discovery support for [Vultr](https://www.vultr.com/). See [these docs](./sd_configs.md#vultr_sd_configs) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041). -* FEATURE: [vmagent](./vmagent.md): allow specifying `-remoteWrite.disableOnDiskQueue` command-line flag per each `-remoteWrite.url`. If multiple `-remoteWrite.disableOnDiskQueue` command-line flags are configured, then the `-remoteWrite.dropSamplesOnOverload` is automatically set to true, so samples are automatically dropped if they cannot be sent to the corresponding `-remoteWrite.url` in a timely manner. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065). Thanks to @rbizos for implementation! -* FEATURE: [vmagent](./vmagent.md): add `path` and `url` labels to `vmagent_remotewrite_push_failures_total` and `vmagent_remotewrite_samples_dropped_total` [metrics](./vmagent.md#monitoring), so the number of failed pushes and dropped samples can be tracked per each `-remoteWrite.url`. -* FEATURE: [stream aggregation](./stream-aggregation.md): add [rate_sum](./stream-aggregation.md#rate_sum) and [rate_avg](./stream-aggregation.md#rate_avg) aggregation outputs. -* FEATURE: [stream aggregation](./stream-aggregation.md): reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402). -* FEATURE: [vmalert](./vmalert.md): add `-datasource.idleConnTimeout`, `-remoteWrite.idleConnTimeout` and `-remoteRead.idleConnTimeout` flags. These flags are set to 50s by default and should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmalert logs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. -* FEATURE: [vmalert](./vmalert.md): add auto request retry for trivial network errors, such as `broken pipe` and `connection reset` for requests to `remoteRead`, `remoteWrite` and `datasource` URLs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. -* FEATURE: [vmalert](./vmalert.md): reduce CPU usage when evaluating high number of alerting and recording rules. -* FEATURE: [vmalert](./vmalert.md): speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6210). -* FEATURE: [vmalert](./vmalert.md): support reading [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) records in `-datasource.url`, `-remoteWrite.url` and `-remoteRead.url` command-line option. For example, `-remoteWrite.url=http://srv+victoria-metrics` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). -* FEATURE: [vmbackup](./vmbackup.md), [vmrestore](./vmrestore.md), [vmbackupmanager](./vmbackupmanager.md): add `-s3TLSInsecureSkipVerify` command-line flag for skipping TLS certificates verification when connecting to S3 endpoint. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support aggregation and deduplication configs before replicating data to configured `-remoteWrite.url` destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add service discovery support for [Vultr](https://www.vultr.com/). See [these docs](https://docs.victoriametrics.com/sd_configs/#vultr_sd_configs) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying `-remoteWrite.disableOnDiskQueue` command-line flag per each `-remoteWrite.url`. If multiple `-remoteWrite.disableOnDiskQueue` command-line flags are configured, then the `-remoteWrite.dropSamplesOnOverload` is automatically set to true, so samples are automatically dropped if they cannot be sent to the corresponding `-remoteWrite.url` in a timely manner. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065). Thanks to @rbizos for implementation! +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `path` and `url` labels to `vmagent_remotewrite_push_failures_total` and `vmagent_remotewrite_samples_dropped_total` [metrics](https://docs.victoriametrics.com/vmagent/#monitoring), so the number of failed pushes and dropped samples can be tracked per each `-remoteWrite.url`. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [rate_sum](https://docs.victoriametrics.com/stream-aggregation/#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/#rate_avg) aggregation outputs. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-datasource.idleConnTimeout`, `-remoteWrite.idleConnTimeout` and `-remoteRead.idleConnTimeout` flags. These flags are set to 50s by default and should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmalert logs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add auto request retry for trivial network errors, such as `broken pipe` and `connection reset` for requests to `remoteRead`, `remoteWrite` and `datasource` URLs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): reduce CPU usage when evaluating high number of alerting and recording rules. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6210). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support reading [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) records in `-datasource.url`, `-remoteWrite.url` and `-remoteRead.url` command-line option. For example, `-remoteWrite.url=http://srv+victoria-metrics` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/), [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): add `-s3TLSInsecureSkipVerify` command-line flag for skipping TLS certificates verification when connecting to S3 endpoint. * FEATURE: expose metric `vm_indexdb_items_dropped_total` to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in `reason` label of the exposed metric. This change also comes with a new [alerting rule](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml) to track changes of this metric. * FEATURE: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): add new alerting rules `TooLongLabelValues` and `TooLongLabelNames` to notify about truncation of label values or names respectively. -* FEATURE: [stream aggregation](./stream-aggregation.md): expose `vm_streamaggr_ignored_samples_total` [counters](./keyConcepts.md#counter) at [`/metrics` page](./README.md#monitoring), which can be used for detecting amount of too old or NaN valued ignored samples. Expose also `vm_streamaggr_samples_lag_seconds` [histogram](./keyConcepts.md#histogram) to monitor aggregated samples lag. -* FEATURE: [stream aggregation](./stream-aggregation.md): improve filtering speed of the received data samples if [match](./stream-aggregation.md#stream-aggregation-config) field is matching only [metric name](./keyConcepts.md#structure-of-a-metric). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): adds `-relabelConfigCheckInterval` flag and `/-/reload` endpoint for reloading relabeling rules. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3923) for details. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): expose `vm_streamaggr_ignored_samples_total` [counters](https://docs.victoriametrics.com/keyconcepts/#counter) at [`/metrics` page](https://docs.victoriametrics.com/#monitoring), which can be used for detecting amount of too old or NaN valued ignored samples. Expose also `vm_streamaggr_samples_lag_seconds` [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) to monitor aggregated samples lag. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): improve filtering speed of the received data samples if [match](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) field is matching only [metric name](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): adds `-relabelConfigCheckInterval` flag and `/-/reload` endpoint for reloading relabeling rules. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3923) for details. -* BUGFIX: [vmui](./README.md#vmui): fix bug that prevents the first query trace from expanding on click event. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186). The issue was introduced in [v1.100.0](./CHANGELOG.md#v11000) release. -* BUGFIX: [vmui](./README.md#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). -* BUGFIX: [vmui](./README.md#vmui): remove redundant requests on the `Explore Cardinality` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240). -* BUGFIX: [vmui](./README.md#vmui): fix handling of URL params for browser history navigation (back and forward buttons). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6126) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232). -* BUGFIX: [vmagent](./vmagent.md): prevent potential panic during [stream aggregation](./stream-aggregation.md) when more than one `-remoteWrite.url` command-line flags are passed to `vmagent` together with non-zero `-remoteWrite.streamAggr.dedupInterval` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205). -* BUGFIX: [vmagent](./vmagent.md): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). -* BUGFIX: [stream aggregation](./stream-aggregation.md): set correct suffix `_prometheus` for aggregation outputs [increase_prometheus](./stream-aggregation.md#increase_prometheus) and [total_prometheus](./stream-aggregation.md#total_prometheus). Before, outputs `total` and `total_prometheus` or `increase` and `increase_prometheus` had the same suffix. -* BUGFIX: [stream aggregation](./stream-aggregation.md): prevent from excessive resource usage when stream aggregation config file is empty. -* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](./MetricsQL.md#aggregate-functions)([`rollup_func[d]`](./MetricsQL.md#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](./CHANGELOG_2022.md#v1830). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): correctly apply `-inmemoryDataFlushInterval` when it's set to minimum supported value 1s. -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vminsert` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): reduce the default value for `-maxLabelValueLen` command-line flag from `16KiB` to `1KiB`. This should prevent from issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176) when time series with too long labels are ingested into VictoriaMetrics. -* BUGFIX: [vmauth](./vmauth.md): properly release memory used for metrics during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix bug that prevents the first query trace from expanding on click event. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186). The issue was introduced in [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000) release. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): remove redundant requests on the `Explore Cardinality` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling of URL params for browser history navigation (back and forward buttons). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6126) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent potential panic during [stream aggregation](https://docs.victoriametrics.com/stream-aggregation.html) when more than one `-remoteWrite.url` command-line flags are passed to `vmagent` together with non-zero `-remoteWrite.streamAggr.dedupInterval` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): set correct suffix `_prometheus` for aggregation outputs [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus). Before, outputs `total` and `total_prometheus` or `increase` and `increase_prometheus` had the same suffix. +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent from excessive resource usage when stream aggregation config file is empty. +* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](https://docs.victoriametrics.com/metricsql/#aggregate-functions)([`rollup_func[d]`](https://docs.victoriametrics.com/metricsql/#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](https://docs.victoriametrics.com/changelog_2022/#v1830). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): correctly apply `-inmemoryDataFlushInterval` when it's set to minimum supported value 1s. +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): reduce the default value for `-maxLabelValueLen` command-line flag from `16KiB` to `1KiB`. This should prevent from issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176) when time series with too long labels are ingested into VictoriaMetrics. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly release memory used for metrics during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247). * BUGFIX: [dashboards](https://grafana.com/orgs/victoriametrics): fix `AnnotationQueryRunner` error in Grafana when executing annotations query against Prometheus backend. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6309) for details. -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): filter deleted label names and values from [`/api/v1/labels`](./url-examples.md#apiv1labels) and [`/api/v1/label/.../values`](./url-examples.md#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](./CHANGELOG_2022.md#v1810). -* BUGFIX: [vmalert-tool](./vmalert-tool.md): fix float values template in `input_series`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6391). -* BUGFIX: [vmalert](./vmalert.md): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): filter deleted label names and values from [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](https://docs.victoriametrics.com/changelog_2022/#v1810). +* BUGFIX: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/): fix float values template in `input_series`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6391). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. * BUGFIX: retry files delete attempts on vXFS file system for `EEXIST` error type. See the [related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396). Thanks to @pludov for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6398). -* DEPRECATION: [vmagent](./vmagent.md): removed deprecated `-remoteWrite.multitenantURL` flag from vmagent. This flag was deprecated since [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0). Use `-enableMultitenantHandlers` instead, as it is easier to use and combine with [multitenant URL at vminsert](./Cluster-VictoriaMetrics.md#multitenancy-via-labels). See [these docs](./vmagent.md#multitenancy) for details. +* DEPRECATION: [vmagent](https://docs.victoriametrics.com/vmagent/): removed deprecated `-remoteWrite.multitenantURL` flag from vmagent. This flag was deprecated since [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0). Use `-enableMultitenantHandlers` instead, as it is easier to use and combine with [multitenant URL at vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels). See [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy) for details. ## [v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0) Released at 2024-04-26 -* FEATURE: [MetricsQL](./MetricsQL.md): support filtering by multiple numeric constants via `q == (C1, ..., CN)` and `q != (C1, ..., CN)` syntax. For example, `status_code == (200, 201, 300)` returns `status_code` metrics with any of `200`, `201` or `300` values, while `status_code != (400, 404, 500)` returns `status_code` metrics with all the values except of `400`, `404` and `500`. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add support for fault domain awareness to `vmselect`. It can be configured to return full responses if up to `-globalReplicationFactor - 1` fault domains (aka `vmstorage` groups) are unavailable. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054) and [these docs](./Cluster-VictoriaMetrics.md#vmstorage-groups-at-vmselect). -* FEATURE: all VictoriaMetrics [enterprise](./enterprise.md) components: add support for automatic issuing of TLS certificates for HTTPS server at `-httpListenAddr` via [Let's Encrypt service](https://letsencrypt.org/). See [these docs](./README.md#automatic-issuing-of-tls-certificates) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949). -* FEATURE: [vmsingle](./Single-Server-VictoriaMetrics.md): support disable or log [implicit conversions](./MetricsQL.md#implicit-query-conversions) for subquery with cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338). -* FEATURE: [vmagent](./vmagent.md): support data replication additionally to sharding among remote storage systems if `-remoteWrite.shardByURLReplicas=N` command-line flag is set additionally to `-remoteWrite.shardByURL` command-line flag, where `N` is desired replication factor. This allows setting up data replication among failure domains when the replication factor is smaller than the number of failure domains. See [these docs](./vmagent.md#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054). -* FEATURE: [vmagent](./vmagent.md): reduce CPU usage when [sharding among remote storage systems](./vmagent.md#sharding-among-remote-storages) is enabled. -* FEATURE: [stream aggregation](./stream-aggregation.md): reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data. -* FEATURE: [vmagent](./vmagent.md): support [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) addresses in `-remoteWrite.url` command-line option and in scrape target urls. For example, `-remoteWrite.url=http://srv+victoria-metrics/api/v1/write` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See [these docs](./vmagent.md#srv-urls) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). -* FEATURE: [stream aggregation](./stream-aggregation.md): allow skipping first N aggregation intervals via cmd-line flag `-streamAggr.ignoreFirstIntervals` for [Single-node VictoriaMetrics](./README.md) or `-remoteWrite.streamAggr.ignoreFirstIntervals` for [vmagent](./vmagent.md). See more details [here](./stream-aggregation.md#ignore-aggregation-intervals-on-start). -* FEATURE: [vmauth](./vmauth.md): support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside `url_prefix` urls in the form `http://srv+addr/path`, where the `addr` is the [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) address, which is automatically resolved to hostnames with TCP ports. See [these docs](./vmauth.md#srv-urls) for details. -* FEATURE: [vmauth](./vmauth.md): support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See [these docs](./vmauth.md#backend-tls-setup). -* FEATURE: [vmauth](./vmauth.md): support regex matching when routing incoming requests based on HTTP [query args](https://en.wikipedia.org/wiki/Query_string) via `src_query_args` option at `url_map`. See [these docs](./vmauth.md#generic-http-proxy-for-different-backends) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070). -* FEATURE: [vmui](./README.md#vmui): optimize auto-suggestion performance for metric names when the database contains big number of unique time series. -* FEATURE: [vmui](./README.md#vmui): in the Select component, user-entered values are now preserved on blur if they match options in the list. -* FEATURE: [vmui](./README.md#vmui): auto-suggestion triggers at any cursor position in the query input. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5864). -* FEATURE: [vmui](./README.md#vmui): update error messages on the Query page for enhanced clarity. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6177). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): support filtering by multiple numeric constants via `q == (C1, ..., CN)` and `q != (C1, ..., CN)` syntax. For example, `status_code == (200, 201, 300)` returns `status_code` metrics with any of `200`, `201` or `300` values, while `status_code != (400, 404, 500)` returns `status_code` metrics with all the values except of `400`, `404` and `500`. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add support for fault domain awareness to `vmselect`. It can be configured to return full responses if up to `-globalReplicationFactor - 1` fault domains (aka `vmstorage` groups) are unavailable. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054) and [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect). +* FEATURE: all VictoriaMetrics [enterprise](https://docs.victoriametrics.com/enterprise/) components: add support for automatic issuing of TLS certificates for HTTPS server at `-httpListenAddr` via [Let's Encrypt service](https://letsencrypt.org/). See [these docs](https://docs.victoriametrics.com/#automatic-issuing-of-tls-certificates) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949). +* FEATURE: [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/): support disable or log [implicit conversions](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions) for subquery with cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support data replication additionally to sharding among remote storage systems if `-remoteWrite.shardByURLReplicas=N` command-line flag is set additionally to `-remoteWrite.shardByURL` command-line flag, where `N` is desired replication factor. This allows setting up data replication among failure domains when the replication factor is smaller than the number of failure domains. See [these docs](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU usage when [sharding among remote storage systems](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) is enabled. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) addresses in `-remoteWrite.url` command-line option and in scrape target urls. For example, `-remoteWrite.url=http://srv+victoria-metrics/api/v1/write` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See [these docs](https://docs.victoriametrics.com/vmagent/#srv-urls) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow skipping first N aggregation intervals via cmd-line flag `-streamAggr.ignoreFirstIntervals` for [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) or `-remoteWrite.streamAggr.ignoreFirstIntervals` for [vmagent](https://docs.victoriametrics.com/vmagent/). See more details [here](https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside `url_prefix` urls in the form `http://srv+addr/path`, where the `addr` is the [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) address, which is automatically resolved to hostnames with TCP ports. See [these docs](https://docs.victoriametrics.com/vmauth/#srv-urls) for details. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See [these docs](https://docs.victoriametrics.com/vmauth/#backend-tls-setup). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support regex matching when routing incoming requests based on HTTP [query args](https://en.wikipedia.org/wiki/Query_string) via `src_query_args` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): optimize auto-suggestion performance for metric names when the database contains big number of unique time series. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): in the Select component, user-entered values are now preserved on blur if they match options in the list. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): auto-suggestion triggers at any cursor position in the query input. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5864). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): update error messages on the Query page for enhanced clarity. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6177). -* BUGFIX: [downsampling](./README.md#downsampling): skip unnecessary index lookups if downsampling wasn't set for ENT versions of VictoriaMetrics. Before, users of VictoriaMetrics ENT could have experience elevated CPU usage even if no downsampling was configured. The issue was introduced in [v1.100.0](./CHANGELOG.md#v11000). -* BUGFIX: [downsampling](./README.md#downsampling): properly populate downsampling metadata for data parts created by VictoriaMetrics ENT versions lower than v1.100.0. The bug could trigger the downsampling actions for parts that were downsampled already. This bug doesn't have any negative effect apart from spending extra CPU resources on the repeated downsampling. The issue was introduced in [v1.100.0](./CHANGELOG.md#v11000). -* BUGFIX: [vmalert](./vmalert.md): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). -* BUGFIX: [vmalert](./vmalert.md): avoid blocking `/api/v1/rules`, `/api/v1/alerts`, `/metrics` APIs when alerting rule uses template functions `query`, which could takes a while to execute. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6079). -* BUGFIX: [vmalert](./vmalert.md): fix links with anchors in vmalert's UI. Starting from [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0) vmalert could ignore anchors pointing to specific rule groups if `search` param was present in URL. -* BUGFIX: [vmauth](./vmauth.md): don't treat concurrency limit hit as an error of the backend. Previously, hitting the concurrency limit would increment both `vmauth_concurrent_requests_limit_reached_total` and `vmauth_user_request_backend_errors_total` counters. Now, only concurrency limit counter is incremented. Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). +* BUGFIX: [downsampling](https://docs.victoriametrics.com/#downsampling): skip unnecessary index lookups if downsampling wasn't set for ENT versions of VictoriaMetrics. Before, users of VictoriaMetrics ENT could have experience elevated CPU usage even if no downsampling was configured. The issue was introduced in [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000). +* BUGFIX: [downsampling](https://docs.victoriametrics.com/#downsampling): properly populate downsampling metadata for data parts created by VictoriaMetrics ENT versions lower than v1.100.0. The bug could trigger the downsampling actions for parts that were downsampled already. This bug doesn't have any negative effect apart from spending extra CPU resources on the repeated downsampling. The issue was introduced in [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): avoid blocking `/api/v1/rules`, `/api/v1/alerts`, `/metrics` APIs when alerting rule uses template functions `query`, which could takes a while to execute. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6079). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix links with anchors in vmalert's UI. Starting from [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0) vmalert could ignore anchors pointing to specific rule groups if `search` param was present in URL. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): don't treat concurrency limit hit as an error of the backend. Previously, hitting the concurrency limit would increment both `vmauth_concurrent_requests_limit_reached_total` and `vmauth_user_request_backend_errors_total` counters. Now, only concurrency limit counter is incremented. Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). ## [v1.100.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.100.1) Released at 2024-04-11 -**Update note 1: This release contains the issue which could lead to extra CPU usage of storage component in ENT distribution of VictoriaMetrics. The issue is caused by [downsampling per distinct sets of time series](./README.md#downsampling) feature introduced in 1.100.0 ENT version. Please, upgrade to [v1.101.0](./CHANGELOG.md#v11010) or rollback to [v1.98.0](./CHANGELOG.md#v1980) ENT version if you're affected.** +**Update note 1: This release contains the issue which could lead to extra CPU usage of storage component in ENT distribution of VictoriaMetrics. The issue is caused by [downsampling per distinct sets of time series](https://docs.victoriametrics.com/#downsampling) feature introduced in 1.100.0 ENT version. Please, upgrade to [v1.101.0](https://docs.victoriametrics.com/changelog/#v11010) or rollback to [v1.98.0](https://docs.victoriametrics.com/changelog/#v1980) ENT version if you're affected.** -**Update note 2: When upgrading to this release from [v1.99.0](./CHANGELOG.md#v1990) or [v1.100.0](./CHANGELOG.md#v11000) it is recommended to reset caches stored on disk according to [these](./Single-Server-VictoriaMetrics.md#cache-removal) docs.** +**Update note 2: When upgrading to this release from [v1.99.0](https://docs.victoriametrics.com/changelog/#v1990) or [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000) it is recommended to reset caches stored on disk according to [these](https://docs.victoriametrics.com/single-server-victoriametrics/#cache-removal) docs.** -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): allow specifying custom backup interval via `-backupInterval` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966). +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): allow specifying custom backup interval via `-backupInterval` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966). * BUGFIX: properly register new entries in IndexDB when many new time series are ingested into VictoriaMetrics in a short period of time. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6069) issues. The bug has been introduced in [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes. +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes. ## [v1.100.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.100.0) Released at 2024-04-04 -**This release contains [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959), which can prevent from storing data for new time series under high rate of search queries. Please rollback to [v1.98.0](./CHANGELOG.md#v1980) or upgrade to [v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0).** +**This release contains [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959), which can prevent from storing data for new time series under high rate of search queries. Please rollback to [v1.98.0](https://docs.victoriametrics.com/changelog/#v1980) or upgrade to [v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0).** -**Update note 1: the `-datasource.lookback` command-line flag at `vmalert` is no-op starting from this release. This flag will be removed in the future, so please switch to [`eval_delay` option](./vmalert.md#groups). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155) for more details.** +**Update note 1: the `-datasource.lookback` command-line flag at `vmalert` is no-op starting from this release. This flag will be removed in the future, so please switch to [`eval_delay` option](https://docs.victoriametrics.com/vmalert/#groups). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155) for more details.** * SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See [the list of issues addressed in Go1.22.1](https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved) and [the list of issues addressed in Go1.22.2](https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved). -* FEATURE: [downsampling](./README.md#downsampling): add ability to configure distinct downsampling per distinct sets of time series and/or [tenants](./Cluster-VictoriaMetrics.md#multitenancy). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960) and [these docs](./README.md#downsampling). -* FEATURE: [vmauth](./vmauth.md): allow discovering ip addresses for backend instances hidden behind a shared hostname, via `discover_backend_ips: true` option. This allows evenly spreading load among backend instances. See [these docs](./vmauth.md#discovering-backend-ips) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707). -* FEATURE: [vmauth](./vmauth.md): allow routing incoming requests based on HTTP [query args](https://en.wikipedia.org/wiki/Query_string) via `src_query_args` option at `url_map`. See [these docs](./vmauth.md#generic-http-proxy-for-different-backends) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878). -* FEATURE: [vmauth](./vmauth.md): allow routing incoming requests based on HTTP request headers via `src_headers` option at `url_map`. See [these docs](./vmauth.md#generic-http-proxy-for-different-backends). -* FEATURE: [vmauth](./vmauth.md): add ability to read auth tokens from arbitrary HTTP request headers via `-httpAuthHeader` command-line flag. Previously auth tokens were read only from `Authorization` HTTP request header. See [these docs](./vmauth.md#reading-auth-tokens-from-other-http-headers) for details. -* FEATURE: [vmauth](./vmauth.md): add ability to authorize by opaque HTTP request header value via `auth_token` option in [-auth.config](./vmauth.md#auth-config). -* FEATURE: [stream aggregation](./stream-aggregation.md): reduce memory usage by up to 5x when aggregating over big number of unique [time series](./keyConcepts.md#time-series). The memory usage reduction is most visible when [stream deduplication](./stream-aggregation.md#deduplication) is enabled. -* FEATURE: [stream aggregation](./stream-aggregation.md): allow using `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line flags without the need to specify `-streamAggr.config` and `-remoteWrite.streamAggr.config`. See [these docs](./stream-aggregation.md#deduplication). -* FEATURE: [stream aggregation](./stream-aggregation.md): add `-streamAggr.dropInputLabels` command-line flag, which can be used for dropping the listed labels from input samples before applying stream [de-duplication](./stream-aggregation.md#deduplication) and aggregation. This is faster and easier to use alternative to [input_relabel_configs](./stream-aggregation.md#relabeling). See [these docs](./stream-aggregation.md#dropping-unneeded-labels). -* FEATURE: [stream aggregation](./stream-aggregation.md): add `dedup_interval` option, which allows configuring individual [deduplication intervals](./stream-aggregation.md#deduplication) per each [stream aggregation config](./stream-aggregation.md#stream-aggregation-config). -* FEATURE: [stream aggregation](./stream-aggregation.md): use the same logic in [stream deduplication](./stream-aggregation.md#deduplication) as in [the deduplication at VictoriaMetrics](./README.md#deduplication). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643). -* FEATURE: [stream aggregation](./stream-aggregation.md): ignore out of order samples when calculating [`increase`](./stream-aggregation.md#increase), [`increase_prometheus`](./stream-aggregation.md#increase_prometheus), [`total`](./stream-aggregation.md#total) and [`total_prometheus`](./stream-aggregation.md#total_prometheus) outputs. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931). -* FEATURE: [stream aggregation](./stream-aggregation.md): add an ability to ignore input samples with old timestamps outside the current aggregation interval. See [these docs](./stream-aggregation.md#ignoring-old-samples) for details. -* FEATURE: [stream aggregation](./stream-aggregation.md): add `keep_metric_names` option, which can be set at [stream aggregation config](./stream-aggregation.md#stream-aggregation-config) in order to keep the original metric names in the output aggregated samples instead of using [the default output metric naming scheme](./stream-aggregation.md#output-metric-names). -* FEATURE: [stream aggregation](./stream-aggregation.md): align the time of aggregated data flush to the specified aggregation `interval`. For example, if `interval` is set to `1m`, then the aggregated data will be flushed at the end of every minute. The alignment can be disabled by setting `no_align_flush_to_interval: true` option at [stream aggregation config](./stream-aggregation.md#stream-aggregation-config). See [these docs](./stream-aggregation.md#flush-time-alignment) for details. -* FEATURE: [stream aggregation](./stream-aggregation.md): add [unique_samples](./stream-aggregation.md#unique_samples) output, which can be used for calculating the number of unique sample values over the given `interval`. -* FEATURE: [stream aggregation](./stream-aggregation.md): add [increase_prometheus](./stream-aggregation.md#increase_prometheus) and [total_prometheus](./stream-aggregation.md#total_prometheus) outputs, which can be used for `increase` and `total` aggregations when the first sample of every new [time series](./keyConcepts.md#time-series) must be ignored. -* FEATURE: [stream aggregation](./stream-aggregation.md): expose `vm_streamaggr_flush_timeouts_total` and `vm_streamaggr_dedup_flush_timeouts_total` [counters](./keyConcepts.md#counter) at [`/metrics` page](./README.md#monitoring), which can be used for detecting flush timeouts for stream aggregation states. Expose also `vm_streamaggr_flush_duration_seconds` and `vm_streamaggr_dedup_flush_duration_seconds` [histograms](./keyConcepts.md#histogram) for monitoring the real flush durations of stream aggregation states. -* FEATURE: [vmui](./README.md#vmui): improve trace display for better visual separation of branches. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5926). -* FEATURE: [vmagent](./vmagent.md): ability to limit the ingestion rate via `-maxIngestionRate` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900). -* FEATURE: [vmagent](./vmagent.md): use the provided `-remoteWrite.tlsServerName` as `Host` header in requests to `-remoteWrite.url`. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802). -* FEATURE: [vmagent](./vmagent.md): add `-remoteWrite.shardByURL.ignoreLabels` command-line flag, which can be used for specifying the ignored list of labels when [sharding by `-remoteWrite.url` is enabled](./vmagent.md#sharding-among-remote-storages). Thanks to @edma2 for the idea and [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938). -* FEATURE: [vmagent](./vmagent.md): automatically reload updated root CA certificates from files without the need to restart `vmagent`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526). -* FEATURE: optimize [`/api/v1/labels`](./url-examples.md#apiv1labels) and [`/api/v1/label/.../values`](./url-examples.md#apiv1labelvalues) when `match[]` filters contains metric name. For example, `/api/v1/label/instance/values?match[]=up` now works much faster than before. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). -* FEATURE: [vmagent](./vmagent.md): accept Prometheus remote write protocol data at `/api/v1/push` endpoint. This simplifies writing data to `vmagent` from Grafana Mimir. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5990). -* FEATURE: [vmctl](./vmctl.md): support client-side TLS configuration for [native protocol](./vmctl.md#migrating-data-from-victoriametrics). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5824). -* FEATURE: [vmctl](./vmctl.md): support client-side TLS configuration for VictoriaMetrics destination specified via `--vm-*` cmd-line flags used in [InfluxDB](./vmctl.md#migrating-data-from-influxdb-1x), [Remote Read protocol](./vmctl.md#migrating-data-by-remote-read-protocol), [OpenTSDB](./vmctl.md#migrating-data-from-opentsdb), [Prometheus](./vmctl.md#migrating-data-from-prometheus) and [Promscale](./vmctl.md#migrating-data-from-promscale) migration modes. -* FEATURE: [vmctl](./vmctl.md): split [explore phase](./vmctl.md#migrating-data-from-victoriametrics) in `vm-native` mode by time intervals when [--vm-native-step-interval](./vmctl.md#using-time-based-chunking-of-migration) is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5369). -* FEATURE: [vmgateway](./vmgateway.md): add `-logInvalidAuthTokens` command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029). -* FEATURE: [graphite](./README.md#graphite-render-api-usage): add support for [aggregateSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.aggregateSeriesLists), [diffSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.diffSeriesLists), [multiplySeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.multiplySeriesLists) and [sumSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.sumSeriesLists) functions. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5809). -* FEATURE: [OpenTelemetry](./README.md#sending-data-via-opentelemetry): add `-opentelemetry.usePrometheusNaming` command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See [these docs](./README.md#sending-data-via-opentelemetry) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037). +* FEATURE: [downsampling](https://docs.victoriametrics.com/#downsampling): add ability to configure distinct downsampling per distinct sets of time series and/or [tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960) and [these docs](https://docs.victoriametrics.com/#downsampling). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow discovering ip addresses for backend instances hidden behind a shared hostname, via `discover_backend_ips: true` option. This allows evenly spreading load among backend instances. See [these docs](https://docs.victoriametrics.com/vmauth/#discovering-backend-ips) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow routing incoming requests based on HTTP [query args](https://en.wikipedia.org/wiki/Query_string) via `src_query_args` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow routing incoming requests based on HTTP request headers via `src_headers` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to read auth tokens from arbitrary HTTP request headers via `-httpAuthHeader` command-line flag. Previously auth tokens were read only from `Authorization` HTTP request header. See [these docs](https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers) for details. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to authorize by opaque HTTP request header value via `auth_token` option in [-auth.config](https://docs.victoriametrics.com/vmauth/#auth-config). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce memory usage by up to 5x when aggregating over big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). The memory usage reduction is most visible when [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) is enabled. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow using `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line flags without the need to specify `-streamAggr.config` and `-remoteWrite.streamAggr.config`. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#deduplication). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `-streamAggr.dropInputLabels` command-line flag, which can be used for dropping the listed labels from input samples before applying stream [de-duplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) and aggregation. This is faster and easier to use alternative to [input_relabel_configs](https://docs.victoriametrics.com/stream-aggregation/#relabeling). See [these docs](https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `dedup_interval` option, which allows configuring individual [deduplication intervals](https://docs.victoriametrics.com/stream-aggregation/#deduplication) per each [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): use the same logic in [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) as in [the deduplication at VictoriaMetrics](https://docs.victoriametrics.com/#deduplication). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): ignore out of order samples when calculating [`increase`](https://docs.victoriametrics.com/stream-aggregation/#increase), [`increase_prometheus`](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus), [`total`](https://docs.victoriametrics.com/stream-aggregation/#total) and [`total_prometheus`](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus) outputs. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add an ability to ignore input samples with old timestamps outside the current aggregation interval. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples) for details. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `keep_metric_names` option, which can be set at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) in order to keep the original metric names in the output aggregated samples instead of using [the default output metric naming scheme](https://docs.victoriametrics.com/stream-aggregation/#output-metric-names). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): align the time of aggregated data flush to the specified aggregation `interval`. For example, if `interval` is set to `1m`, then the aggregated data will be flushed at the end of every minute. The alignment can be disabled by setting `no_align_flush_to_interval: true` option at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). See [these docs](https://docs.victoriametrics.com/stream-aggregation/#flush-time-alignment) for details. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [unique_samples](https://docs.victoriametrics.com/stream-aggregation/#unique_samples) output, which can be used for calculating the number of unique sample values over the given `interval`. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus) outputs, which can be used for `increase` and `total` aggregations when the first sample of every new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) must be ignored. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): expose `vm_streamaggr_flush_timeouts_total` and `vm_streamaggr_dedup_flush_timeouts_total` [counters](https://docs.victoriametrics.com/keyconcepts/#counter) at [`/metrics` page](https://docs.victoriametrics.com/#monitoring), which can be used for detecting flush timeouts for stream aggregation states. Expose also `vm_streamaggr_flush_duration_seconds` and `vm_streamaggr_dedup_flush_duration_seconds` [histograms](https://docs.victoriametrics.com/keyconcepts/#histogram) for monitoring the real flush durations of stream aggregation states. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve trace display for better visual separation of branches. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5926). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): ability to limit the ingestion rate via `-maxIngestionRate` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): use the provided `-remoteWrite.tlsServerName` as `Host` header in requests to `-remoteWrite.url`. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-remoteWrite.shardByURL.ignoreLabels` command-line flag, which can be used for specifying the ignored list of labels when [sharding by `-remoteWrite.url` is enabled](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages). Thanks to @edma2 for the idea and [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): automatically reload updated root CA certificates from files without the need to restart `vmagent`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526). +* FEATURE: optimize [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) when `match[]` filters contains metric name. For example, `/api/v1/label/instance/values?match[]=up` now works much faster than before. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): accept Prometheus remote write protocol data at `/api/v1/push` endpoint. This simplifies writing data to `vmagent` from Grafana Mimir. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5990). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): support client-side TLS configuration for [native protocol](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5824). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): support client-side TLS configuration for VictoriaMetrics destination specified via `--vm-*` cmd-line flags used in [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x), [Remote Read protocol](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol), [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb), [Prometheus](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus) and [Promscale](https://docs.victoriametrics.com/vmctl/#migrating-data-from-promscale) migration modes. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): split [explore phase](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics) in `vm-native` mode by time intervals when [--vm-native-step-interval](https://docs.victoriametrics.com/vmctl/#using-time-based-chunking-of-migration) is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5369). +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add `-logInvalidAuthTokens` command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029). +* FEATURE: [graphite](https://docs.victoriametrics.com/#graphite-render-api-usage): add support for [aggregateSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.aggregateSeriesLists), [diffSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.diffSeriesLists), [multiplySeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.multiplySeriesLists) and [sumSeriesLists](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.sumSeriesLists) functions. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5809). +* FEATURE: [OpenTelemetry](https://docs.victoriametrics.com/#sending-data-via-opentelemetry): add `-opentelemetry.usePrometheusNaming` command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See [these docs](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037). * BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. The probability of this bug has been increased significantly after [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0) because of optimizations related to registering new time series. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959) issue. -* BUGFIX: [vmagent](./vmagent.md): properly set `Host` header in requests to scrape targets if it is specified via [`headers` option](./sd_configs.md#http-api-client-options). Thanks to @fholzer for [the bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969) and [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970). -* BUGFIX: [vmagent](./vmagent.md): properly set `Host` header in requests to scrape targets when [`server_name` option at `tls_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) is set. Previously the `Host` header was set incorrectly to the target hostname in this case. -* BUGFIX: do not drop `match[]` filter at [`/api/v1/series`](./url-examples.md#apiv1series) if `-search.ignoreExtraFiltersAtLabelsAPI` command-line flag is set, since missing `match[]` filter breaks `/api/v1/series` requests. -* BUGFIX: [vmagent](./vmagent.md): return proper responses for [AWS Firehose](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) requests according to [these docs](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6016) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037). -* BUGFIX: [vmctl](./vmctl.md): properly parse TLS key and CA files for [InfluxDB](./vmctl.md#migrating-data-from-influxdb-1x) and [OpenTSDB](./vmctl.md#migrating-data-from-opentsdb) migration modes. -* BUGFIX: [vmui](./README.md#vmui): fix VictoriaLogs UI query handling to correctly apply `_time` filter across all queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920). -* BUGFIX: [vmui](./README.md#vmui): fix issue where `step` parameter wasn't updated after moving the focus outside the `step` input field in Firefox. -* BUGFIX: [vmui](./README.md#vmui): fix freezing when pasting a query with autocomplete enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5923). -* BUGFIX: [vmui](./README.md#vmui): fix auto-suggestion trigger issue after whitespace char. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5866). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with `-search.maxLabelsAPIDuration` duration. Before, `-search.maxExportDuration` value was used by mistake. The bug has been introduced in [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0). Thanks to @kbweave for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5992). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `Host` header in requests to scrape targets if it is specified via [`headers` option](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). Thanks to @fholzer for [the bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969) and [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `Host` header in requests to scrape targets when [`server_name` option at `tls_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) is set. Previously the `Host` header was set incorrectly to the target hostname in this case. +* BUGFIX: do not drop `match[]` filter at [`/api/v1/series`](https://docs.victoriametrics.com/url-examples/#apiv1series) if `-search.ignoreExtraFiltersAtLabelsAPI` command-line flag is set, since missing `match[]` filter breaks `/api/v1/series` requests. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): return proper responses for [AWS Firehose](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) requests according to [these docs](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6016) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): properly parse TLS key and CA files for [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x) and [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb) migration modes. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix VictoriaLogs UI query handling to correctly apply `_time` filter across all queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix issue where `step` parameter wasn't updated after moving the focus outside the `step` input field in Firefox. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix freezing when pasting a query with autocomplete enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5923). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix auto-suggestion trigger issue after whitespace char. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5866). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with `-search.maxLabelsAPIDuration` duration. Before, `-search.maxExportDuration` value was used by mistake. The bug has been introduced in [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0). Thanks to @kbweave for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5992). * BUGFIX: properly wait for force merge to be completed during the shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5944) for the details. -* BUGFIX: [vmalert](./vmalert.md): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. -* BUGFIX: [vmalert](./vmalert.md): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. -* BUGFIX: [vmalert](./vmalert.md): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). ## [v1.99.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0) Released at 2024-03-01 -**This release contains [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959), which can prevent from storing data for new time series under high rate of search queries. Please rollback to [v1.98.0](./CHANGELOG.md#v1980) or upgrade to [v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0).** +**This release contains [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959), which can prevent from storing data for new time series under high rate of search queries. Please rollback to [v1.98.0](https://docs.victoriametrics.com/changelog/#v1980) or upgrade to [v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0).** -* FEATURE: [MetricsQL](./MetricsQL.md): propagate [label filters](./keyConcepts.md#filtering) via all the [label manipulation functions](./MetricsQL.md#label-manipulation-functions). For example, `label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"}` is now transformed to `label_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}`. This should reduce the amounts of time series processed during query execution. -* FEATURE: [MetricsQL](./MetricsQL.md): add [count_values_over_time](./MetricsQL.md#count_values_over_time) function. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847). -* FEATURE: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): expose `vm_last_partition_parts` [metrics](./README.md#monitoring), which show the number of [parts in the latest partition](./README.md#storage). These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition. -* FEATURE: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): expose `vm_snapshots` [metric](./README.md#monitoring), which shows the current number of snapshots created via [snapshot API](./README.md#how-to-work-with-snapshots). -* FEATURE: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `-search.ignoreExtraFiltersAtLabelsAPI` command-line flag, which can be used for reducing load on VictoriaMetrics when [/api/v1/labels](./url-examples.md#apiv1labels), [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) or [/api/v1/series](./url-examples.md#apiv1series) are queried with too broad [`extra_filters` or `extra_label`](./README.md#prometheus-querying-api-enhancements), which match many time series. See [these docs](./README.md#resource-usage-limits) for details. -* FEATURE: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): allow limiting CPU and RAM usage at [/api/v1/labels](./url-examples.md#apiv1labels), [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) and [/api/v1/series](./url-examples.md#apiv1series) on systems with [high churn rate](./FAQ.md#what-is-high-churn-rate) via `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries` command-line flags. See [these docs](./README.md#resource-usage-limits) for details. -* FEATURE: [vmagent](./vmagent.md): support reading [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) metrics in [OpenTelemetry](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-100.html) format from [Amazon Data Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). -* FEATURE: [vmagent](./vmagent.md): add support for `client_id` option into [kuma_sd_configs](./sd_configs.md#kuma_sd_configs) in the same way as Prometheus does. See [this pull request](https://github.com/prometheus/prometheus/pull/13278). -* FEATURE: [vmagent](./vmagent.md): add support for `enable_compression` option in [scrape_configs](./sd_configs.md#scrape_configs) in order to be compatible with Prometheus scrape configs. See [this pull request](https://github.com/prometheus/prometheus/pull/13166) and [this feature request](https://github.com/prometheus/prometheus/issues/12319). Note that `vmagent` was always supporting [`disable_compression` option](./vmagent.md#scrape_config-enhancements) before Prometheus added `enable_compression` option. -* FEATURE: [vmctl](./vmctl.md): support client-side TLS configuration for [InfluxDB](./vmctl.md#migrating-data-from-influxdb-1x), [Remote Read protocol](./vmctl.md#migrating-data-by-remote-read-protocol) and [OpenTSDB](./vmctl.md#migrating-data-from-opentsdb). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for pull requests [1](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5783), [2](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5798), [3](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5797). -* FEATURE: [vmui](./README.md#vmui): preserve [`WITH` templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs) when clicking the `prettify query` button at the right side of query input field. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5383). -* FEATURE: [vmui](./README.md#vmui): allow filling gaps on graphs with interpolated lines as Grafana does. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862). -* FEATURE: [vmalert](./README.md#vmalert): support filtering by group, rule or labels in [vmalert's UI](./vmalert.md#web) for `/groups` and `/alerts` pages. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5791) by @victoramsantos. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): propagate [label filters](https://docs.victoriametrics.com/keyconcepts/#filtering) via all the [label manipulation functions](https://docs.victoriametrics.com/metricsql/#label-manipulation-functions). For example, `label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"}` is now transformed to `label_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}`. This should reduce the amounts of time series processed during query execution. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [count_values_over_time](https://docs.victoriametrics.com/metricsql/#count_values_over_time) function. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847). +* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): expose `vm_last_partition_parts` [metrics](https://docs.victoriametrics.com/#monitoring), which show the number of [parts in the latest partition](https://docs.victoriametrics.com/#storage). These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition. +* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): expose `vm_snapshots` [metric](https://docs.victoriametrics.com/#monitoring), which shows the current number of snapshots created via [snapshot API](https://docs.victoriametrics.com/#how-to-work-with-snapshots). +* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `-search.ignoreExtraFiltersAtLabelsAPI` command-line flag, which can be used for reducing load on VictoriaMetrics when [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) are queried with too broad [`extra_filters` or `extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements), which match many time series. See [these docs](https://docs.victoriametrics.com/#resource-usage-limits) for details. +* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): allow limiting CPU and RAM usage at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) and [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) on systems with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) via `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries` command-line flags. See [these docs](https://docs.victoriametrics.com/#resource-usage-limits) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support reading [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) metrics in [OpenTelemetry](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-100.html) format from [Amazon Data Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `client_id` option into [kuma_sd_configs](https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs) in the same way as Prometheus does. See [this pull request](https://github.com/prometheus/prometheus/pull/13278). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `enable_compression` option in [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) in order to be compatible with Prometheus scrape configs. See [this pull request](https://github.com/prometheus/prometheus/pull/13166) and [this feature request](https://github.com/prometheus/prometheus/issues/12319). Note that `vmagent` was always supporting [`disable_compression` option](https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements) before Prometheus added `enable_compression` option. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): support client-side TLS configuration for [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x), [Remote Read protocol](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol) and [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for pull requests [1](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5783), [2](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5798), [3](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5797). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): preserve [`WITH` templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs) when clicking the `prettify query` button at the right side of query input field. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5383). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): allow filling gaps on graphs with interpolated lines as Grafana does. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862). +* FEATURE: [vmalert](https://docs.victoriametrics.com/#vmalert): support filtering by group, rule or labels in [vmalert's UI](https://docs.victoriametrics.com/vmalert/#web) for `/groups` and `/alerts` pages. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5791) by @victoramsantos. * FEATURE: [docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#docker-compose-environment-for-victoriametrics): create a separate [docker-compose environment](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose-victorialogs.yml) for VictoriaLogs installation, including fluentbit and [VictoriaLogs Grafana datasource](https://github.com/VictoriaMetrics/victorialogs-datasource). See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#victorialogs-server) for details. -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): wait for up 30 seconds before making a [snapshot](./README.md#how-to-work-with-snapshots) for backup if `vmstorage` is temporarily unavailable. This should prevent from `vmbackupmanager` termination in this case. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5859). +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): wait for up 30 seconds before making a [snapshot](https://docs.victoriametrics.com/#how-to-work-with-snapshots) for backup if `vmstorage` is temporarily unavailable. This should prevent from `vmbackupmanager` termination in this case. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5859). -* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](./vmagent.md). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. +* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](https://docs.victoriametrics.com/vmagent/). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. * BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to `-httpListenAddr` every 2 minutes, which was disabled in [v1.98.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.98.0) when addressing [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450) for details on why the periodic closing of incoming connections has been returned back. -* BUGFIX: [vmagent](./vmagent.md): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). -* BUGFIX: [vmagent](./vmagent.md): fix possible deadlock when [sharding among remote storages](./vmagent.md#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). -* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](./keyConcepts.md#instant-query) or [/api/v1/query_range](./keyConcepts.md#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. -* BUGFIX: [vmalert](./README.md#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. -* BUGFIX: [vmui](./README.md#vmui): fix Popper display issues. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5842). -* BUGFIX: [vmselect](./README.md): format time with milliseconds precision when `__timestamp__:rfc3339` is specified for [/api/v1/export/csv](./README.md#how-to-export-csv-data) API response. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5837) for details. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly construct the destination label in [label_join](./MetricsQL.md#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible deadlock when [sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). +* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) or [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix Popper display issues. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5842). +* BUGFIX: [vmselect](https://docs.victoriametrics.com/): format time with milliseconds precision when `__timestamp__:rfc3339` is specified for [/api/v1/export/csv](https://docs.victoriametrics.com/#how-to-export-csv-data) API response. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5837) for details. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly construct the destination label in [label_join](https://docs.victoriametrics.com/metricsql/#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. ## [v1.98.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.98.0) @@ -312,869 +312,869 @@ Released at 2024-02-14 * SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See [the list of issues addressed in Go1.21.7](https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved), plus [the changelog for Go1.22.0](https://go.dev/doc/go1.22). -* FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at `-httpListenAddr` (aka [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)). See [these docs](./README.md#mtls-protection) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5458). +* FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at `-httpListenAddr` (aka [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)). See [these docs](https://docs.victoriametrics.com/#mtls-protection) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5458). * FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple `-httpListenAddr` command-line flags. For example, `./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428` starts VictoriaMetrics, which accepts incoming http requests at both `some-host:12345` and `localhost:8428`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470). * FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, `-remoteWrite.sendTimeout=',20s,'` specifies three `-remoteWrite.sendTimeout` values - the first one and the last one have default values (`30s` in this case), while the second one is set to `20s`. -* FEATURE: [vmauth](./vmauth.md): add support for [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See [these docs](./vmauth.md#mtls-based-request-routing) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1547). -* FEATURE: [vmagent](./vmagent.md) and [single-node VictoriaMetrics](./README.md): add support for data ingestion via [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/) aka `/api/beta/sketches` endpoint. See [these docs](./README.md#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3091). Thanks to @AndrewChubatiuk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584). -* FEATURE: [vmagent](./vmagent.md): add `-remoteWrite.tlsHandshakeTimeout` command-line flag for tuning the timeout needed for establishing TLS connections to `-remoteWrite.url`. Bigger tls handshake timeouts should reduce the probability of `http: TLS handshake error from ...: EOF` errors at the remote storage side under high load. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `-disableReroutingOnUnavailable` command-line flag to `vminsert`, which can be used for reducing resource usage spikes at `vmstorage` nodes during rolling restart. See [these docs](./Cluster-VictoriaMetrics.md#improving-re-routing-performance-during-restart). Thanks to @Muxa1L for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713). -* FEATURE: add `-search.resetRollupResultCacheOnStartup` command-line flag for resetting [query cache](./README.md#rollup-result-cache) on startup. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834). -* FEATURE: [MetricsQL](./MetricsQL.md): propagate label filters across [label_set](./MetricsQL.md#label_set) and [alias](./MetricsQL.md#alias) functions. For example, `label_set(q1, "a", "b") + q2{c="d"}` is automatically transformed to `label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"}` now. This should improve performance for such queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827#issuecomment-1654095358). -* FEATURE: [MetricsQL](./MetricsQL.md): add [sum_eq_over_time](./MetricsQL.md#sum_eq_over_time), [sum_gt_over_time](./MetricsQL.md#sum_gt_over_time) and [sum_le_over_time](./MetricsQL.md#sum_le_over_time) functions. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add support for [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See [these docs](https://docs.victoriametrics.com/vmauth/#mtls-based-request-routing) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1547). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com): add support for data ingestion via [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/) aka `/api/beta/sketches` endpoint. See [these docs](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3091). Thanks to @AndrewChubatiuk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-remoteWrite.tlsHandshakeTimeout` command-line flag for tuning the timeout needed for establishing TLS connections to `-remoteWrite.url`. Bigger tls handshake timeouts should reduce the probability of `http: TLS handshake error from ...: EOF` errors at the remote storage side under high load. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `-disableReroutingOnUnavailable` command-line flag to `vminsert`, which can be used for reducing resource usage spikes at `vmstorage` nodes during rolling restart. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart). Thanks to @Muxa1L for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713). +* FEATURE: add `-search.resetRollupResultCacheOnStartup` command-line flag for resetting [query cache](https://docs.victoriametrics.com/#rollup-result-cache) on startup. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): propagate label filters across [label_set](https://docs.victoriametrics.com/metricsql/#label_set) and [alias](https://docs.victoriametrics.com/metricsql/#alias) functions. For example, `label_set(q1, "a", "b") + q2{c="d"}` is automatically transformed to `label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"}` now. This should improve performance for such queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827#issuecomment-1654095358). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [sum_eq_over_time](https://docs.victoriametrics.com/metricsql/#sum_eq_over_time), [sum_gt_over_time](https://docs.victoriametrics.com/metricsql/#sum_gt_over_time) and [sum_le_over_time](https://docs.victoriametrics.com/metricsql/#sum_le_over_time) functions. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641). * FEATURE: [dashboards/vmagent](https://grafana.com/grafana/dashboards/12683): add `Targets scraped/s` stat panel showing the number of targets scraped by the vmagent per-second. * FEATURE: [dashboards/all](https://grafana.com/orgs/victoriametrics): add new panel `CPU spent on GC`. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed. -* FEATURE: [vmalert](./README.md#vmalert): support [filtering](https://prometheus.io/docs/prometheus/2.49/querying/api/#rules) for `/api/v1/rules` API. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5749) by @victoramsantos. -* FEATURE: [vmbackup](./vmbackup.md): support client-side TLS configuration for creating and deleting snapshots via `-snapshot.tls*` cmd-line flags. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5724). Thanks to @khushijain21 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5738). -* FEATURE: [vmui](./README.md#vmui): add a time picker to the "Logs Explorer" page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5673). +* FEATURE: [vmalert](https://docs.victoriametrics.com/#vmalert): support [filtering](https://prometheus.io/docs/prometheus/2.49/querying/api/#rules) for `/api/v1/rules` API. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5749) by @victoramsantos. +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): support client-side TLS configuration for creating and deleting snapshots via `-snapshot.tls*` cmd-line flags. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5724). Thanks to @khushijain21 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5738). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add a time picker to the "Logs Explorer" page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5673). * BUGFIX: all VictoriaMetrics components: do not close connections to `-httpListenAddr` every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037). If you still need periodically closing client connections because of some reason, then pass the desired timeout to `-http.connTimeout` command-line flag. -* BUGFIX: [vmagent](./vmagent.md): reduce CPU usage when `-promscrape.dropOriginalLabels` command-line flag is set. This issue has been introduced in [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) when addressing [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389). -* BUGFIX: [vmauth](./vmauth.md): properly release memory during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690). -* BUGFIX: [vmauth](./vmauth.md): properly expose `vmauth_unauthorized_user_concurrent_requests_capacity`, `vmauth_unauthorized_user_concurrent_requests_current`, `vmauth_user_concurrent_requests_capacity` and `vmauth_user_concurrent_requests_current` [metrics](./vmauth.md#monitoring) after [config reload](./vmauth.md#config-reload). Previously these metrics didn't work after config reload. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly propagate [label filters](./keyConcepts.md#filtering) from multiple arguments passed to [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle precision errors when calculating [changes](./MetricsQL.md#changes), [changes_prometheus](./MetricsQL.md#changes_prometheus), [increases_over_time](./MetricsQL.md#increases_over_time) and [resets](./MetricsQL.md#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). -* BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from [`/-/reload` endpoint](./vmagent.md#configuration-update). Previously [single-node VictoriaMetrics](./README.md) was returning 204 http status code. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774). -* BUGFIX: properly store [staleness markers](./vmagent.md#prometheus-staleness-markers) for [self-scraped metrics](./README.md#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU usage when `-promscrape.dropOriginalLabels` command-line flag is set. This issue has been introduced in [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) when addressing [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly release memory during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly expose `vmauth_unauthorized_user_concurrent_requests_capacity`, `vmauth_unauthorized_user_concurrent_requests_current`, `vmauth_user_concurrent_requests_capacity` and `vmauth_user_concurrent_requests_current` [metrics](https://docs.victoriametrics.com/vmauth/#monitoring) after [config reload](https://docs.victoriametrics.com/vmauth/#config-reload). Previously these metrics didn't work after config reload. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly propagate [label filters](https://docs.victoriametrics.com/keyconcepts/#filtering) from multiple arguments passed to [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle precision errors when calculating [changes](https://docs.victoriametrics.com/metricsql/#changes), [changes_prometheus](https://docs.victoriametrics.com/metricsql/#changes_prometheus), [increases_over_time](https://docs.victoriametrics.com/metricsql/#increases_over_time) and [resets](https://docs.victoriametrics.com/metricsql/#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). +* BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from [`/-/reload` endpoint](https://docs.victoriametrics.com/vmagent/#configuration-update). Previously [single-node VictoriaMetrics](https://docs.victoriametrics.com/) was returning 204 http status code. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774). +* BUGFIX: properly store [staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for [self-scraped metrics](https://docs.victoriametrics.com/#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). * BUGFIX: prevent from possible `too big indexBlockSize` panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics. -* BUGFIX: [vmui](./README.md#vmui): fix the graph dragging for Firefox and Safari. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5764). -* BUGFIX: [vmui](./README.md#vmui): fix handling invalid timezone. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5732). -* BUGFIX: [vmui](./README.md#vmui): fix the bug where the select does not open. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5728). -* BUGFIX: [vmui](./README.md#vmui): clear entered text in select after selecting a value. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5727). -* BUGFIX: [vmui](./README.md#vmui): improve the operation of the context for autocomplete. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5736), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5737) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5739) issues. -* BUGFIX: [dashboards](https://grafana.com/orgs/victoriametrics): update `Storage full ETA` panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of [deduplication](./README.md#deduplication) in VictoriaMetrics it rather produced skewed results. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5747). -* BUGFIX: [vmalert](./README.md#vmalert): reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the graph dragging for Firefox and Safari. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5764). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling invalid timezone. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5732). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the bug where the select does not open. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5728). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): clear entered text in select after selecting a value. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5727). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): improve the operation of the context for autocomplete. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5736), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5737) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5739) issues. +* BUGFIX: [dashboards](https://grafana.com/orgs/victoriametrics): update `Storage full ETA` panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of [deduplication](https://docs.victoriametrics.com/#deduplication) in VictoriaMetrics it rather produced skewed results. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5747). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/#vmalert): reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy. ## [v1.97.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.6) Released at 2024-07-17 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/CHANGELOG.html#v1970) release** * SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in [Go1.22.5](https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See [alpine 3.20.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html). -* BUGFIX: [vmagent](./vmagent.md): fix panic when using multiple topics with the same name when [ingesting metrics from Kafka](./vmagent.md#kafka-integration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636) for the details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix panic when using multiple topics with the same name when [ingesting metrics from Kafka](https://docs.victoriametrics.com/vmagent/#kafka-integration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636) for the details. * BUGFIX: all VictoriaMetrics components: properly calculate `process_resident_memory_bytes` metric for OS with non-default value of memory page size. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457) for details. -* BUGFIX: [vmalert](./vmalert.md): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. -* BUGFIX: [vmbackup](./vmbackup.md): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. -* BUGFIX: [vmalert](./vmalert.md) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. -* BUGFIX: [vmalert](./vmalert.md): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). -* BUGFIX: [vmalert](./vmalert.md): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add validation for the max value specified for `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330) for details. -* BUGFIX: [vmui](./README.md#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). -* BUGFIX: [vmalert-tool](./vmalert-tool.md): prevent hanging when processing groups without rules. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500). -* BUGFIX: [vmui](./README.md#vmui): fix input cursor position reset in modal settings. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6530). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [histogram_quantile](./MetricsQL.md#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add validation for the max value specified for `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330) for details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). +* BUGFIX: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/): prevent hanging when processing groups without rules. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix input cursor position reset in modal settings. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6530). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). ## [v1.97.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5) Released at 2024-06-07 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/CHANGELOG.html#v1970) release** * SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in [Go1.22.3](https://github.com/golang/go/issues?q=milestone%3AGo1.22.3+label%3ACherryPickApproved) and [Go1.22.4](https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See [alpine 3.20.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.0-released.html). -* BUGFIX: [vmui](./README.md#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). -* BUGFIX: [vmui](./README.md#vmui): remove redundant requests on the `Explore Cardinality` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240). -* BUGFIX: [vmui](./README.md#vmui): fix handling of URL params for browser history navigation (back and forward buttons). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6126) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232). -* BUGFIX: [vmagent](./vmagent.md): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). -* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](./MetricsQL.md#aggregate-functions)([`rollup_func[d]`](./MetricsQL.md#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](./CHANGELOG_2022.md#v1830). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmstorage` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): correctly apply `-inmemoryDataFlushInterval` when it's set to minimum supported value 1s. -* BUGFIX: [vmauth](./vmauth.md): properly release memory used for metrics during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): filter deleted label names and values from [`/api/v1/labels`](./url-examples.md#apiv1labels) and [`/api/v1/label/.../values`](./url-examples.md#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](./CHANGELOG_2022.md#v1810). -* BUGFIX: [vmalert](./vmalert.md): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): remove redundant requests on the `Explore Cardinality` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling of URL params for browser history navigation (back and forward buttons). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6126) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). +* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](https://docs.victoriametrics.com/metricsql/#aggregate-functions)([`rollup_func[d]`](https://docs.victoriametrics.com/metricsql/#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](https://docs.victoriametrics.com/changelog_2022/#v1830). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): correctly apply `-inmemoryDataFlushInterval` when it's set to minimum supported value 1s. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly release memory used for metrics during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): filter deleted label names and values from [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](https://docs.victoriametrics.com/changelog_2022/#v1810). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. * BUGFIX: retry files delete attempts on vXFS file system for `EEXIST` error type. See the [related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396). Thanks to @pludov for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6398). ## [v1.97.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.4) Released at 2024-04-19 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/changelog/#v1970) release** * SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See [the list of issues addressed in Go1.22.1](https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved) and [the list of issues addressed in Go1.22.2](https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved). * BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959) issue. -* BUGFIX: [vmagent](./vmagent.md): properly set `Host` header in requests to scrape targets when [`server_name` option at `tls_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) is set. Previously the `Host` header was set incorrectly to the target hostname in this case. -* BUGFIX: [vmagent](./vmagent.md): properly set `Host` header in requests to scrape targets if it is specified via [`headers` option](./sd_configs.md#http-api-client-options). Thanks to @fholzer for [the bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969) and [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970). -* BUGFIX: [vmalert](./vmalert.md): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. -* BUGFIX: [vmalert](./vmalert.md): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. -* BUGFIX: [vmalert](./vmalert.md): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). -* BUGFIX: [vmalert](./vmalert.md): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). -* BUGFIX: [vmalert](./vmalert.md): avoid blocking `/api/v1/rules`, `/api/v1/alerts`, `/metrics` APIs when alerting rule uses template functions `query`, which could takes a while to execute. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6079). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `Host` header in requests to scrape targets when [`server_name` option at `tls_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) is set. Previously the `Host` header was set incorrectly to the target hostname in this case. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `Host` header in requests to scrape targets if it is specified via [`headers` option](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). Thanks to @fholzer for [the bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969) and [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): avoid blocking `/api/v1/rules`, `/api/v1/alerts`, `/metrics` APIs when alerting rule uses template functions `query`, which could takes a while to execute. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6079). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes. ## [v1.97.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.3) Released at 2024-03-01 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/changelog/#v1970) release** -* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](./vmagent.md). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. -* BUGFIX: take into account `-search.maxUniqueTimeseries` limit at [/api/v1/labels](./url-examples.md#apiv1labels), [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) and [/api/v1/series](./url-examples.md#apiv1series). This limit was ignored at these endpoints starting from [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0) when trying to address [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). It has been appeared that this can result in high CPU usage when heavyweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return `limit exceeded` error to heavyweight queries instead of burning CPU. +* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](https://docs.victoriametrics.com/vmagent/). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. +* BUGFIX: take into account `-search.maxUniqueTimeseries` limit at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) and [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). This limit was ignored at these endpoints starting from [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0) when trying to address [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). It has been appeared that this can result in high CPU usage when heavyweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return `limit exceeded` error to heavyweight queries instead of burning CPU. * BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to `-httpListenAddr` every 2 minutes, which was disabled in [v1.97.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.2) when addressing [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450) for details on why the periodic closing of incoming connections has been returned back. -* BUGFIX: [vmagent](./vmagent.md): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). -* BUGFIX: [vmagent](./vmagent.md): fix possible deadlock when [sharding among remote storages](./vmagent.md#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). -* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](./keyConcepts.md#instant-query) - or [/api/v1/query_range](./keyConcepts.md#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. -* BUGFIX: [vmalert](./README.md#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. -* BUGFIX: [vmalert](./README.md#vmalert): reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly construct the destination label in [label_join](./MetricsQL.md#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible deadlock when [sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). +* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) + or [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/#vmalert): reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly construct the destination label in [label_join](https://docs.victoriametrics.com/metricsql/#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. ## [v1.97.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.2) Released at 2024-02-14 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/changelog/#v1970) release** * SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See [the list of issues addressed in Go1.21.7](https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved), plus [the changelog for Go1.22.0](https://go.dev/doc/go1.22). * BUGFIX: all VictoriaMetrics components: do not close connections to `-httpListenAddr` every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037). If you still need periodically closing client connections because of some reason, then pass the desired timeout to `-http.connTimeout` command-line flag. -* BUGFIX: [vmagent](./vmagent.md): reduce CPU usage when `-promscrape.dropOriginalLabels` command-line flag is set. This issue has been introduced in [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) when addressing [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389). -* BUGFIX: [vmauth](./vmauth.md): properly release memory during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690). -* BUGFIX: [vmauth](./vmauth.md): properly expose `vmauth_unauthorized_user_concurrent_requests_capacity`, `vmauth_unauthorized_user_concurrent_requests_current`, `vmauth_user_concurrent_requests_capacity` and `vmauth_user_concurrent_requests_current` [metrics](./vmauth.md#monitoring) after [config reload](./vmauth.md#config-reload). Previously these metrics didn't work after config reload. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly propagate [label filters](./keyConcepts.md#filtering) from multiple arguments passed to [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle precision errors when calculating [changes](./MetricsQL.md#changes), [changes_prometheus](./MetricsQL.md#changes_prometheus), [increases_over_time](./MetricsQL.md#increases_over_time) and [resets](./MetricsQL.md#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). -* BUGFIX: properly store [staleness markers](./vmagent.md#prometheus-staleness-markers) for [self-scraped metrics](./README.md#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU usage when `-promscrape.dropOriginalLabels` command-line flag is set. This issue has been introduced in [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) when addressing [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly release memory during config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly expose `vmauth_unauthorized_user_concurrent_requests_capacity`, `vmauth_unauthorized_user_concurrent_requests_current`, `vmauth_user_concurrent_requests_capacity` and `vmauth_user_concurrent_requests_current` [metrics](https://docs.victoriametrics.com/vmauth/#monitoring) after [config reload](https://docs.victoriametrics.com/vmauth/#config-reload). Previously these metrics didn't work after config reload. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly propagate [label filters](https://docs.victoriametrics.com/keyconcepts/#filtering) from multiple arguments passed to [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle precision errors when calculating [changes](https://docs.victoriametrics.com/metricsql/#changes), [changes_prometheus](https://docs.victoriametrics.com/metricsql/#changes_prometheus), [increases_over_time](https://docs.victoriametrics.com/metricsql/#increases_over_time) and [resets](https://docs.victoriametrics.com/metricsql/#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). +* BUGFIX: properly store [staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for [self-scraped metrics](https://docs.victoriametrics.com/#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). * BUGFIX: prevent from possible `too big indexBlockSize` panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics. ## [v1.97.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.1) Released at 2024-02-01 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/changelog/#v1970) release** -* FEATURE: [vmagent](./vmagent.md): add support for `username_file` option at `basic_auth` section in [`scrape_configs`](./sd_configs.md#http-api-client-options). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511). Thanks to @wasim-nihal for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5720). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `username_file` option at `basic_auth` section in [`scrape_configs`](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511). Thanks to @wasim-nihal for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5720). -* BUGFIX: [vmagent](./vmagent.md): fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0). * BUGFIX: fix `runtime error: slice bounds out of range` panic, which can occur during query execution. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5733). The bug has been introduced in `v1.97.0`. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle `avg_over_time({some_filter}[d]) keep_metric_names` queries, where [`some_filter`](./keyConcepts.md#filtering) matches multiple time series with multiple names, while `d` is bigger or equal to `3h`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle `avg_over_time({some_filter}[d]) keep_metric_names` queries, where [`some_filter`](https://docs.victoriametrics.com/keyconcepts/#filtering) matches multiple time series with multiple names, while `d` is bigger or equal to `3h`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556). * BUGFIX: [dashboards/single](https://grafana.com/grafana/dashboards/10229): fix typo in query for `version` annotation which falsely produced many version change events. ## [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0) Released at 2024-01-30 -**v1.97.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](./enterprise.md). +**v1.97.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). -The v1.97.x line will be supported for at least 12 months since [v1.97.0](./CHANGELOG.md#v1970) release** +The v1.97.x line will be supported for at least 12 months since [v1.97.0](https://docs.victoriametrics.com/changelog/#v1970) release** * SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See [alpine 3.19.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html). * SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See [the list of issues addressed in Go1.21.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.6+label%3ACherryPickApproved). -* FEATURE: improve new [time series](./keyConcepts.md#time-series) registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and [implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212). -* FEATURE: make [background merge](./README.md#storage) more responsive and scalable. This should help the following issues: [5190](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190), [3425](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425), [648](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648). -* FEATURE: [graphite](./README.md#graphite-render-api-usage): add support for negative index in `groupByNode` and `aliasByNode` functions. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5581). -* FEATURE: [vmagent](./vmagent.md): add support for discovering [Hetzner Cloud](https://www.hetzner.com/cloud) and [Hetzner Robot](https://docs.hetzner.com/robot) scrape targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154) and [these docs](./sd_configs.md#hetzner_sd_configs). -* FEATURE: [vmagent](./vmagent.md): add support for [DataDog v2 data ingestion protocol](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](./README.md#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451). -* FEATURE: [vmagent](./vmagent.md): reduce initial memory usage when scraping targets, which return huge responses (for example, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable [stream parsing mode](./vmagent.md#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567). -* FEATURE: [vmagent](./vmagent.md): expose ability to set OAuth2 endpoint parameters per each `-remoteWrite.url` via the command-line flag `-remoteWrite.oauth2.endpointParams`. See [these docs](./vmagent.md#advanced-usage). Thanks to @mhill-holoplot for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427). -* FEATURE: [vmagent](./vmagent.md): add ability to set `attach_metadata.node=true` option for all the [`kubernetes_sd_configs`](./sd_configs.md#kubernetes_sd_configs) defined at [`-promscrape.config`](./vmagent.md#quick-start) via `-promscrape.kubernetes.attachNodeMetadataAll` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640). Thanks to @wasim-nihal for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593). -* FEATURE: [vmagent](./vmagent.md): do not send unfinished [aggregation state](./stream-aggregation.md) on shutdown or [hot config reload](./stream-aggregation.md#configuration-update) by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config. See more details [here](./stream-aggregation.md#aggregation-outputs). -* FEATURE: [streaming aggregation](./stream-aggregation.md): expand `%{ENV_VAR}` placeholders in config files with the corresponding environment variable values. -* FEATURE: [vmalert](./vmagent.md): expose ability to set OAuth2 endpoint parameters via the following command-line flags: +* FEATURE: improve new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and [implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212). +* FEATURE: make [background merge](https://docs.victoriametrics.com/#storage) more responsive and scalable. This should help the following issues: [5190](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190), [3425](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425), [648](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648). +* FEATURE: [graphite](https://docs.victoriametrics.com/#graphite-render-api-usage): add support for negative index in `groupByNode` and `aliasByNode` functions. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5581). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for discovering [Hetzner Cloud](https://www.hetzner.com/cloud) and [Hetzner Robot](https://docs.hetzner.com/robot) scrape targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154) and [these docs](https://docs.victoriametrics.com/sd_configs/#hetzner_sd_configs). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [DataDog v2 data ingestion protocol](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce initial memory usage when scraping targets, which return huge responses (for example, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose ability to set OAuth2 endpoint parameters per each `-remoteWrite.url` via the command-line flag `-remoteWrite.oauth2.endpointParams`. See [these docs](https://docs.victoriametrics.com/vmagent/#advanced-usage). Thanks to @mhill-holoplot for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to set `attach_metadata.node=true` option for all the [`kubernetes_sd_configs`](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) defined at [`-promscrape.config`](https://docs.victoriametrics.com/vmagent/#quick-start) via `-promscrape.kubernetes.attachNodeMetadataAll` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640). Thanks to @wasim-nihal for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): do not send unfinished [aggregation state](https://docs.victoriametrics.com/stream-aggregation/) on shutdown or [hot config reload](https://docs.victoriametrics.com/stream-aggregation/#configuration-update) by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config. See more details [here](https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs). +* FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): expand `%{ENV_VAR}` placeholders in config files with the corresponding environment variable values. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmagent/): expose ability to set OAuth2 endpoint parameters via the following command-line flags: * `-datasource.oauth2.endpointParams` for `-datasource.url` * `-notifier.oauth2.endpointParams` for `-notifier.url` * `-remoteRead.oauth2.endpointParams` for `-remoteRead.url` * `-remoteWrite.oauth2.endpointParams` for `-remoteWrite.url` -* FEATURE: [vmauth](./vmauth.md): add ability to proxy incoming requests to different backends based on the requested host via `src_hosts` option at `url_map`. See [these docs](./vmauth.md#generic-http-proxy-for-different-backends). -* FEATURE: [vmauth](./vmauth.md): expose `vmauth_user_request_backend_errors_total` and `vmauth_unauthorized_user_request_backend_errors_total` [metrics](./vmauth.md#monitoring), which track the number of failed requests because of backend errors. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). -* FEATURE: [vmauth](./vmauth.md): add an ability to specify additional labels for [per-user metrics](./vmauth.md#monitoring) via `metric_labels` section. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). -* FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at `-httpListenAddr` already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by [vmctl](./vmctl.md). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645). -* FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to `-pushmetrics.url`. See [`push metrics` docs](./README.md#push-metrics). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to proxy incoming requests to different backends based on the requested host via `src_hosts` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): expose `vmauth_user_request_backend_errors_total` and `vmauth_unauthorized_user_request_backend_errors_total` [metrics](https://docs.victoriametrics.com/vmauth/#monitoring), which track the number of failed requests because of backend errors. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add an ability to specify additional labels for [per-user metrics](https://docs.victoriametrics.com/vmauth/#monitoring) via `metric_labels` section. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565). +* FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at `-httpListenAddr` already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by [vmctl](https://docs.victoriametrics.com/vmctl/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645). +* FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to `-pushmetrics.url`. See [`push metrics` docs](https://docs.victoriametrics.com/#push-metrics). * FEATURE: all VictoriaMetrics components: add `-metrics.exposeMetadata` command-line flag, which allows displaying `TYPE` and `HELP` metadata at `/metrics` page exposed at `-httpListenAddr`. This may be needed when the `/metrics` page is scraped by collector, which requires the `TYPE` and `HELP` metadata such as [Google Cloud Managed Prometheus](https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type). -* FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using `file:///path/to/file` or `http://host/path` syntax for the following command-line flags: `-configAuthKey`, `-deleteAuthKey`, `-flagsAuthKey`, `-forceMergeAuthKey`, `-forceFlushAuthKey`, `-httpAuth.password`, `-metricsAuthKey`, `-pprofAuthKey`, `-reloadAuthKey`, `-search.resetCacheAuthKey`, `-snapshotAuthKey`. For example, `-httpAuth.password=file:///path/to/password`. See [these docs](./README.md#security) for details. +* FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using `file:///path/to/file` or `http://host/path` syntax for the following command-line flags: `-configAuthKey`, `-deleteAuthKey`, `-flagsAuthKey`, `-forceMergeAuthKey`, `-forceFlushAuthKey`, `-httpAuth.password`, `-metricsAuthKey`, `-pprofAuthKey`, `-reloadAuthKey`, `-search.resetCacheAuthKey`, `-snapshotAuthKey`. For example, `-httpAuth.password=file:///path/to/password`. See [these docs](https://docs.victoriametrics.com/#security) for details. * FEATURE: [dashboards/cluster](https://grafana.com/grafana/dashboards/11176): add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components. * FEATURE: [dashboards/cluster](https://grafana.com/grafana/dashboards/11176): update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation. * FEATURE: [dashboards/single](https://grafana.com/grafana/dashboards/10229): change dashboard title from `VictoriaMetrics` to `VictoriaMetrics - single-node`. The new title should provide better understanding of this dashboard purpose. -* FEATURE: [vmctl](./vmctl.md): rename cmd-line flag `vm-native-disable-retries` to `vm-native-disable-per-metric-migration` to better reflect its meaning. -* FEATURE: [vmctl](./vmctl.md): add `-vm-native-src-insecure-skip-verify` and `-vm-native-dst-insecure-skip-verify` command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): rename cmd-line flag `vm-native-disable-retries` to `vm-native-disable-per-metric-migration` to better reflect its meaning. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `-vm-native-src-insecure-skip-verify` and `-vm-native-dst-insecure-skip-verify` command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses. * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): add `job` label to `DiskRunsOutOfSpace` alerting rule, so it is easier to understand to which installation the triggered instance belongs. * FEATURE: `vmstorage`: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by `-maxLabelsPerTimeseries` or `-maxLabelValueLen` command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong. -* FEATURE: [vmui](./README.md#vmui): add the ability to export and import query reports: +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to export and import query reports: * add a `Query Analyzer` page that allows you to build graphs from `JSON` data containing the results of executing a query request. * add an `Export query` button to the graph that saves the result of executing the query in `JSON`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5497). -* FEATURE: [vmui](./README.md#vmui): add `-vmui.defaultTimezone` flag to set a default timezone. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375) and [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#timezone-configuration). -* FEATURE: [vmui](./README.md#vmui): include UTC in the timezone selection dropdown for standardized time referencing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375). -* FEATURE: [vmui](./README.md#vmui): add the ability to expand/collapse all tracing entries and download tracing data in .json format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5677). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `-vmui.defaultTimezone` flag to set a default timezone. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375) and [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#timezone-configuration). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): include UTC in the timezone selection dropdown for standardized time referencing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to expand/collapse all tracing entries and download tracing data in .json format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5677). * FEATURE: add [VictoriaMetrics datasource](https://github.com/VictoriaMetrics/victoriametrics-datasource) to docker compose environment. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5363). -* BUGFIX: properly return the list of matching label names and label values from [`/api/v1/labels`](./url-examples.md#apiv1labels) and [`/api/v1/label/.../values`](./url-examples.md#apiv1labelvalues) when the database contains more than `-search.maxUniqueTimeseries` unique [time series](./keyConcepts.md#time-series) on the selected time range. Previously VictoriaMetrics could return `the number of matching timeseries exceeds ...` error in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). -* BUGFIX: properly return errors from [export APIs](./README.md#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly return full results when `-search.skipSlowReplicas` command-line flag is passed to `vmselect` and when [vmstorage groups](./Cluster-VictoriaMetrics.md#vmstorage-groups-at-vmselect) are in use. Previously partial results could be returned in this case. -* BUGFIX: `vminsert`: properly accept samples via [OpenTelemetry data ingestion protocol](./README.md#sending-data-via-opentelemetry) when these samples have no [resource attributes](https://opentelemetry.io/docs/instrumentation/go/resources/). Previously such samples were silently skipped. -* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). -* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](./FAQ.md#what-is-high-churn-rate). This could result in increasing CPU load over time. -* BUGFIX: [vmalert](./vmalert.md): check `-external.url` schema when starting vmalert, must be `http` or `https`. Before, alertmanager could reject alert notifications if `-external.url` contained no or wrong schema. -* BUGFIX: [vmalert](./vmalert.md): automatically add `exported_` prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5161). -* BUGFIX: [vmalert](./vmalert.md): autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0`. Previously, `ALERTS_FOR_STATE` was generated only for alerts with `for > 0`. The change aligns with Prometheus behavior. See more details in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5648). -* BUGFIX: [vmalert](./vmalert.md): fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with `context cancelled` error. -* BUGFIX: [MetricsQL](./MetricsQL.md): consistently sort results for `q1 or q2` query, so they do not change colors with each refresh in Grafana. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5393). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly return results from [bottomk](./MetricsQL.md#bottomk) and `bottomk_*()` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle queries, which wrap [rollup functions](./MetricsQL.md#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). -* BUGFIX: [vmctl](./vmctl.md): retry on import errors in `vm-native` mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. -* BUGFIX: [vmagent](./vmagent.md): properly assume role with [AWS IRSA authorization](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Previously role chaining was not supported. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822) for details. -* BUGFIX: [vmagent](./vmagent.md): exit if there is config syntax error in [`scrape_config_files`](./vmagent.md#loading-scrape-configs-from-multiple-files) when `-promscrape.config.strictParse=true`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5508). -* BUGFIX: [vmagent](./vmagent.md): properly discover targets for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). Previously some `endpoints` and `endpointslice` targets could be left undiscovered or some targets could have missing `__meta_*` labels when performing service discovery in busy Kubernetes clusters with large number of pods. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557). -* BUGFIX: [vmagent](./vmagent.md): respect explicitly set `series_limit: 0` in [scrape_config](./sd_configs.md#scrape_configs). This allows removing [`series_limit` restriction](./vmagent.md#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. -* BUGFIX: [vmagent](./vmagent.md): do not discover scrape targets for already terminated pods and containers in [`kubernetes_sd_configs`](./sd_configs.md#kubernetes_sd_configs). Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them. -* BUGFIX: [vmagent](./vmagent.md): properly follow redirects to another hostnames when `follow_redirects: true` option is set in [`scrape_configs`](./sd_configs.md#http-api-client-options) while [stream parsing mode](./vmagent.md#stream-parsing-mode) is disabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931). -* BUGFIX: [vmui](./README.md#vmui): fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5460). -* BUGFIX: [vmui](./README.md#vmui): fix the display of autocomplete results and cache the results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5472) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5470). -* BUGFIX: [vmui](./README.md#vmui): send `step` param for instant queries. The change reverts [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896) due to reasons explained in [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896#issuecomment-1896704401). -* BUGFIX: [vmui](./README.md#vmui): fix the `Enter` key press issue in the `Query` field when using autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5667). -* BUGFIX: all: fix potential panic during components shutdown when [metrics push](./README.md#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). +* BUGFIX: properly return the list of matching label names and label values from [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) when the database contains more than `-search.maxUniqueTimeseries` unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) on the selected time range. Previously VictoriaMetrics could return `the number of matching timeseries exceeds ...` error in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055). +* BUGFIX: properly return errors from [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly return full results when `-search.skipSlowReplicas` command-line flag is passed to `vmselect` and when [vmstorage groups](https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect) are in use. Previously partial results could be returned in this case. +* BUGFIX: `vminsert`: properly accept samples via [OpenTelemetry data ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) when these samples have no [resource attributes](https://opentelemetry.io/docs/instrumentation/go/resources/). Previously such samples were silently skipped. +* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). +* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). This could result in increasing CPU load over time. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): check `-external.url` schema when starting vmalert, must be `http` or `https`. Before, alertmanager could reject alert notifications if `-external.url` contained no or wrong schema. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): automatically add `exported_` prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5161). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0`. Previously, `ALERTS_FOR_STATE` was generated only for alerts with `for > 0`. The change aligns with Prometheus behavior. See more details in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5648). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with `context cancelled` error. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): consistently sort results for `q1 or q2` query, so they do not change colors with each refresh in Grafana. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5393). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly return results from [bottomk](https://docs.victoriametrics.com/metricsql/#bottomk) and `bottomk_*()` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle queries, which wrap [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): retry on import errors in `vm-native` mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly assume role with [AWS IRSA authorization](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Previously role chaining was not supported. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): exit if there is config syntax error in [`scrape_config_files`](https://docs.victoriametrics.com/vmagent/#loading-scrape-configs-from-multiple-files) when `-promscrape.config.strictParse=true`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5508). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover targets for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously some `endpoints` and `endpointslice` targets could be left undiscovered or some targets could have missing `__meta_*` labels when performing service discovery in busy Kubernetes clusters with large number of pods. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): respect explicitly set `series_limit: 0` in [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). This allows removing [`series_limit` restriction](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not discover scrape targets for already terminated pods and containers in [`kubernetes_sd_configs`](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly follow redirects to another hostnames when `follow_redirects: true` option is set in [`scrape_configs`](https://docs.victoriametrics.com/sd_configs/#http-api-client-options) while [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is disabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5460). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the display of autocomplete results and cache the results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5472) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5470). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): send `step` param for instant queries. The change reverts [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896) due to reasons explained in [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896#issuecomment-1896704401). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the `Enter` key press issue in the `Query` field when using autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5667). +* BUGFIX: all: fix potential panic during components shutdown when [metrics push](https://docs.victoriametrics.com/#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). * BUGFIX: `vmselect`: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached. -* BUGFIX: `vmselect`: set proper timestamps in [instant query](./keyConcepts.md#instant-query) results if the query contains [rollup functions](./MetricsQL.md#rollup-functions) with lookbehind windows in square brackets exceeding 3 hours. For example, `sum(avg_over_time(up[24h]))`. This bug has been introduced in [v1.95.0](./CHANGELOG_2023.md#v1950). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5730) issues. +* BUGFIX: `vmselect`: set proper timestamps in [instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query) results if the query contains [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) with lookbehind windows in square brackets exceeding 3 hours. For example, `sum(avg_over_time(up[24h]))`. This bug has been introduced in [v1.95.0](https://docs.victoriametrics.com/changelog_2023/#v1950). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5730) issues. ## [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) -See changes [here](./CHANGELOG_2023.md#v1960) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1960) ## [v1.95.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1) -See changes [here](./CHANGELOG_2023.md#v1951) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1951) ## [v1.95.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.0) -See changes [here](./CHANGELOG_2023.md#v1950) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1950) ## [v1.94.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.94.0) -See changes [here](./CHANGELOG_2023.md#v1940) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1940) ## [v1.93.16](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.16) Released at 2024-07-17 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/CHANGELOG.html#v1930) release** * SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in [Go1.22.5](https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See [alpine 3.20.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html). -* BUGFIX: [vmbackup](./vmbackup.md): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. -* BUGFIX: [vmalert](./vmalert.md): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. -* BUGFIX: [vmalert](./vmalert.md): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). -* BUGFIX: [vmalert](./vmalert.md) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. -* BUGFIX: [vmalert](./vmalert.md): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). -* BUGFIX: [vmui](./README.md#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [histogram_quantile](./MetricsQL.md#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix incorrect redirection in WebUI of vmalert. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6620). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/) enterprise: properly configure authentication with S3 when `-s3.configFilePath` cmd-line flag is specified for reading rule configs. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly specify oauth2 `ClientSecret` when configuring authentication for `notifier.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6478). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): **copy row** button in Table view produces unexpected result. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6421) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6495). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) over Prometheus buckets with inconsistent values. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580#issuecomment-2186659102) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547). Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). ## [v1.93.15](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.15) Released at 2024-06-07 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/CHANGELOG.html#v1930) release** * SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in [Go1.22.3](https://github.com/golang/go/issues?q=milestone%3AGo1.22.3+label%3ACherryPickApproved) and [Go1.22.4](https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See [alpine 3.20.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.0-released.html). -* BUGFIX: [vmui](./README.md#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). -* BUGFIX: [vmagent](./vmagent.md): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). -* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](./MetricsQL.md#aggregate-functions)([`rollup_func[d]`](./MetricsQL.md#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](./CHANGELOG_2022.md#v1830). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): filter deleted label names and values from [`/api/v1/labels`](./url-examples.md#apiv1labels) and [`/api/v1/label/.../values`](./url-examples.md#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](./CHANGELOG_2022.md#v1810). -* BUGFIX: [vmalert](./vmalert.md): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). +* BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](https://docs.victoriametrics.com/metricsql/#aggregate-functions)([`rollup_func[d]`](https://docs.victoriametrics.com/metricsql/#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](https://docs.victoriametrics.com/changelog_2022/#v1830). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): filter deleted label names and values from [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) responses when `match[]` filter matches small number of time series. The issue was introduced [v1.81.0](https://docs.victoriametrics.com/changelog_2022/#v1810). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested. * BUGFIX: retry files delete attempts on vXFS file system for `EEXIST` error type. See the [related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396). Thanks to @pludov for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6398). ## [v1.93.14](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14) Released at 2024-04-19 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See [the list of issues addressed in Go1.22.1](https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved) and [the list of issues addressed in Go1.22.2](https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved). -* BUGFIX: [vmalert](./vmalert.md): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. -* BUGFIX: [vmalert](./vmalert.md): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. -* BUGFIX: [vmalert](./vmalert.md): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). -* BUGFIX: [vmalert](./vmalert.md): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): set correct `endsAt` value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5995) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly account for `-rule.resendDelay` for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than `-rule.resendDelay`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6028) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): respect `-remoteWrite.maxBatchSize` at shutdown period. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025). Thanks to @jiekun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6039). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110). ## [v1.93.13](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.13) Released at 2024-03-01 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](./vmagent.md). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. -* BUGFIX: [vmagent](./vmagent.md): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). -* BUGFIX: [vmagent](./vmagent.md): fix possible deadlock when [sharding among remote storages](./vmagent.md#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). +* BUGFIX: downgrade Go builder from `1.22.0` to `1.21.7`, since `1.22.0` contains [the bug](https://github.com/golang/go/issues/65705), which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at [vmagent](https://docs.victoriametrics.com/vmagent/). This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): accept OpenTelemetry data at `/opentelemetry/v1/metrics`, since the `/v1/metrics` suffix is hardcoded at [OpenTelemetry protocol specification](https://opentelemetry.io/docs/specs/otlp/#otlphttp-request). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible deadlock when [sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) is enabled with `-remoteWrite.shardByURL` command-line flag. Thanks to @penguinlav for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834) for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833). -* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](./keyConcepts.md#instant-query) - or [/api/v1/query_range](./keyConcepts.md#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). -* BUGFIX: [Single-node VictoriaMetrics](./README.md) and `vmselect` in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. -* BUGFIX: [vmalert](./README.md#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly construct the destination label in [label_join](./MetricsQL.md#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. +* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) + or [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795). +* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/#vmalert): consistently sort groups by name and filename on `/groups` page in UI. This should prevent non-deterministic sorting for groups with identical names. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly construct the destination label in [label_join](https://docs.victoriametrics.com/metricsql/#label_join) if it is used as source label. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5881) for details. ## [v1.93.12](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.12) Released at 2024-02-14 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See [the list of issues addressed in Go1.21.7](https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved), plus [the changelog for Go1.22.0](https://go.dev/doc/go1.22). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly propagate [label filters](./keyConcepts.md#filtering) from multiple arguments passed to [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle precision errors when calculating [changes](./MetricsQL.md#changes), [changes_prometheus](./MetricsQL.md#changes_prometheus), [increases_over_time](./MetricsQL.md#increases_over_time) and [resets](./MetricsQL.md#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). -* BUGFIX: properly store [staleness markers](./vmagent.md#prometheus-staleness-markers) for [self-scraped metrics](./README.md#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly propagate [label filters](https://docs.victoriametrics.com/keyconcepts/#filtering) from multiple arguments passed to [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum({job="foo"}, {job="bar"}) by (job) + a` was improperly optimized to `sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}` before being executed. This could lead to unexpected results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle precision errors when calculating [changes](https://docs.victoriametrics.com/metricsql/#changes), [changes_prometheus](https://docs.victoriametrics.com/metricsql/#changes_prometheus), [increases_over_time](https://docs.victoriametrics.com/metricsql/#increases_over_time) and [resets](https://docs.victoriametrics.com/metricsql/#resets) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767). +* BUGFIX: properly store [staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for [self-scraped metrics](https://docs.victoriametrics.com/#monitoring) on single-node VictoriaMetrics shutdown. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943). * BUGFIX: prevent from possible `too big indexBlockSize` panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics. ## [v1.93.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.11) Released at 2024-02-01 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See [alpine 3.19.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html). -* BUGFIX: properly return errors from [export APIs](./README.md#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). -* BUGFIX: [vmagent](./vmagent.md): properly discover targets for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). Previously some `endpoints` and `endpointslice` targets could be left undiscovered or some targets could have missing `__meta_*` labels when performing service discovery in busy Kubernetes clusters with large number of pods. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557). -* BUGFIX: [vmagent](./vmagent.md): respect explicitly set `series_limit: 0` in [scrape_config](./sd_configs.md#scrape_configs). This allows removing [`series_limit` restriction](./vmagent.md#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). +* BUGFIX: properly return errors from [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover targets for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously some `endpoints` and `endpointslice` targets could be left undiscovered or some targets could have missing `__meta_*` labels when performing service discovery in busy Kubernetes clusters with large number of pods. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): respect explicitly set `series_limit: 0` in [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). This allows removing [`series_limit` restriction](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). ## [v1.93.10](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.10) Released at 2024-01-17 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See [the list of issues addressed in Go1.21.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.6+label%3ACherryPickApproved). -* BUGFIX: `vminsert`: properly accept samples via [OpenTelemetry data ingestion protocol](./README.md#sending-data-via-opentelemetry) when these samples have no [resource attributes](https://opentelemetry.io/docs/instrumentation/go/resources/). Previously such samples were silently skipped. -* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). -* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](./FAQ.md#what-is-high-churn-rate). This could result in increasing CPU load over time. -* BUGFIX: [vmalert](./vmalert.md): check `-external.url` schema when starting vmalert, must be `http` or `https`. Before, alertmanager could reject alert notifications if `-external.url` contained no or wrong schema. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly return results from [bottomk](./MetricsQL.md#bottomk) and `bottomk_*()` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle queries, which wrap [rollup functions](./MetricsQL.md#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). -* BUGFIX: [vmagent](./vmagent.md): properly assume role with [AWS IRSA authorization](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Previously role chaining was not supported. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822) for details. -* BUGFIX: all: fix potential panic during components shutdown when [metrics push](./README.md#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). -* BUGFIX: [vmctl](./vmctl.md): check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5446). -* BUGFIX: [vmctl](./vmctl.md): retry on import errors in `vm-native` mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. +* BUGFIX: `vminsert`: properly accept samples via [OpenTelemetry data ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) when these samples have no [resource attributes](https://opentelemetry.io/docs/instrumentation/go/resources/). Previously such samples were silently skipped. +* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). +* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). This could result in increasing CPU load over time. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): check `-external.url` schema when starting vmalert, must be `http` or `https`. Before, alertmanager could reject alert notifications if `-external.url` contained no or wrong schema. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly return results from [bottomk](https://docs.victoriametrics.com/metricsql/#bottomk) and `bottomk_*()` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle queries, which wrap [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly assume role with [AWS IRSA authorization](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Previously role chaining was not supported. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822) for details. +* BUGFIX: all: fix potential panic during components shutdown when [metrics push](https://docs.victoriametrics.com/#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5446). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): retry on import errors in `vm-native` mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. ## [v1.93.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.9) -See changes [here](./CHANGELOG_2023.md#v1939) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1939) ## [v1.93.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.8) -See changes [here](./CHANGELOG_2023.md#v1938) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1938) ## [v1.93.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.7) -See changes [here](./CHANGELOG_2023.md#v1937) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1937) ## [v1.93.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.6) -See changes [here](./CHANGELOG_2023.md#v1936) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1936) ## [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) -See changes [here](./CHANGELOG_2023.md#v1935) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1935) ## [v1.93.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.4) -See changes [here](./CHANGELOG_2023.md#v1934) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1934) ## [v1.93.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.3) -See changes [here](./CHANGELOG_2023.md#v1933) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1933) ## [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2) -See changes [here](./CHANGELOG_2023.md#v1932) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1932) ## [v1.93.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.1) -See changes [here](./CHANGELOG_2023.md#v1931) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1931) ## [v1.93.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.0) -See changes [here](./CHANGELOG_2023.md#v1930) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1930) ## [v1.92.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.1) -See changes [here](./CHANGELOG_2023.md#v1921) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1921) ## [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0) -See changes [here](./CHANGELOG_2023.md#v1920) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1920) ## [v1.91.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.3) -See changes [here](./CHANGELOG_2023.md#v1913) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1913) ## [v1.91.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.2) -See changes [here](./CHANGELOG_2023.md#v1912) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1912) ## [v1.91.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1) -See changes [here](./CHANGELOG_2023.md#v1911) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1911) ## [v1.91.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.0) -See changes [here](./CHANGELOG_2023.md#v1910) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1910) ## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0) -See changes [here](./CHANGELOG_2023.md#v1900) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1900) ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1) -See changes [here](./CHANGELOG_2023.md#v1891) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1891) ## [v1.89.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.0) -See changes [here](./CHANGELOG_2023.md#v1890) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1890) ## [v1.88.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.88.1) -See changes [here](./CHANGELOG_2023.md#v1881) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1881) ## [v1.88.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.88.0) -See changes [here](./CHANGELOG_2023.md#v1880) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1880) ## [v1.87.14](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.14) Released at 2024-02-01 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See [alpine 3.19.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html). -* BUGFIX: properly return errors from [export APIs](./README.md#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). -* BUGFIX: [vmagent](./vmagent.md): respect explicitly set `series_limit: 0` in [scrape_config](./sd_configs.md#scrape_configs). This allows removing [`series_limit` restriction](./vmagent.md#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). +* BUGFIX: properly return errors from [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): respect explicitly set `series_limit: 0` in [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). This allows removing [`series_limit` restriction](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571). ## [v1.87.13](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.13) Released at 2024-01-17 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See [the list of issues addressed in Go1.21.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.6+label%3ACherryPickApproved). -* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle queries, which wrap [rollup functions](./MetricsQL.md#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](./MetricsQL.md#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). -* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](./FAQ.md#what-is-high-churn-rate). This could result in increasing CPU load over time. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly return results from [bottomk](./MetricsQL.md#bottomk) and `bottomk_...` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). -* BUGFIX: all: fix potential panic during components shutdown when [metrics push](./README.md#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). +* BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle queries, which wrap [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to @atykhyy for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414). +* BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). This could result in increasing CPU load over time. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly return results from [bottomk](https://docs.victoriametrics.com/metricsql/#bottomk) and `bottomk_...` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to @xiaozongyang for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509). +* BUGFIX: all: fix potential panic during components shutdown when [metrics push](https://docs.victoriametrics.com/#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to @zhdd99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549). # [v1.87.12](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.12) -See changes [here](./CHANGELOG_2023.md#v18712) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v18712) ## [v1.87.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.11) -See changes [here](./CHANGELOG_2023.md#v18711) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v18711) ## [v1.87.10](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.10) -See changes [here](./CHANGELOG_2023.md#v18710) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v18710) ## [v1.87.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.9) -See changes [here](./CHANGELOG_2023.md#v1879) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1879) ## [v1.87.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.8) -See changes [here](./CHANGELOG_2023.md#v1878) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1878) ## [v1.87.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.7) -See changes [here](./CHANGELOG_2023.md#v1877) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1877) ## [v1.87.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.6) -See changes [here](./CHANGELOG_2023.md#v1876) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1876) ## [v1.87.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.5) -See changes [here](./CHANGELOG_2023.md#v1875) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1875) ## [v1.87.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.4) -See changes [here](./CHANGELOG_2023.md#v1874) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1874) ## [v1.87.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.3) -See changes [here](./CHANGELOG_2023.md#v1873) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1873) ## [v1.87.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.2) -See changes [here](./CHANGELOG_2023.md#v1872) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1872) ## [v1.87.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.1) -See changes [here](./CHANGELOG_2023.md#v1871) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1871) ## [v1.87.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.0) -See changes [here](./CHANGELOG_2023.md#v1870) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1870) ## [v1.86.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.2) -See changes [here](./CHANGELOG_2023.md#v1862) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1862) ## [v1.86.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.1) -See changes [here](./CHANGELOG_2023.md#v1861) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1861) ## [v1.86.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.0) -See changes [here](./CHANGELOG_2023.md#v1860) +See changes [here](https://docs.victoriametrics.com/changelog_2023/#v1860) ## [v1.85.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.3) -See changes [here](./CHANGELOG_2022.md#v1853) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1853) ## [v1.85.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.2) -See changes [here](./CHANGELOG_2022.md#v1852) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1852) ## [v1.85.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.1) -See changes [here](./CHANGELOG_2022.md#v1851) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1851) ## [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0) -See changes [here](./CHANGELOG_2022.md#v1850) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1850) ## [v1.84.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.84.0) -See changes [here](./CHANGELOG_2022.md#v1840) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1840) ## [v1.83.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.83.1) -See changes [here](./CHANGELOG_2022.md#v1831) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1831) ## [v1.83.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.83.0) -See changes [here](./CHANGELOG_2022.md#v1830) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1830) ## [v1.82.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.1) -See changes [here](./CHANGELOG_2022.md#v1821) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1821) ## [v1.82.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.0) -See changes [here](./CHANGELOG_2022.md#v1820) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1820) ## [v1.81.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.2) -See changes [here](./CHANGELOG_2022.md#v1812) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1812) ## [v1.81.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.1) -See changes [here](./CHANGELOG_2022.md#v1811) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1811) ## [v1.81.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.0) -See changes [here](./CHANGELOG_2022.md#v1810) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1810) ## [v1.80.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.80.0) -See changes [here](./CHANGELOG_2022.md#v1800) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1800) ## [v1.79.14](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.14) -See changes [here](./CHANGELOG_2022.md#v17914) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v17914) ## [v1.79.13](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.13) -See changes [here](./CHANGELOG_2022.md#v17913) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v17913) ## [v1.79.12](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.12) -See changes [here](./CHANGELOG_2022.md#v17912) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v17912) ## [v1.79.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.11) -See changes [here](./CHANGELOG_2022.md#v17911) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v17911) ## [v1.79.10](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.10) -See changes [here](./CHANGELOG_2022.md#v17910) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v17910) ## [v1.79.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.9) -See changes [here](./CHANGELOG_2022.md#v1799) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1799) ## [v1.79.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.8) -See changes [here](./CHANGELOG_2022.md#v1798) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1798) ## [v1.79.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.7) -See changes [here](./CHANGELOG_2022.md#v1797) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1797) ## [v1.79.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.6) -See changes [here](./CHANGELOG_2022.md#v1796) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1796) ## [v1.79.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.5) -See changes [here](./CHANGELOG_2022.md#v1795) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1795) ## [v1.79.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.4) -See changes [here](./CHANGELOG_2022.md#v1794) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1794) ## [v1.79.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.3) -See changes [here](./CHANGELOG_2022.md#v1793) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1793) ## [v1.79.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.2) -See changes [here](./CHANGELOG_2022.md#v1792) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1792) ## [v1.79.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.1) -See changes [here](./CHANGELOG_2022.md#v1791) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1791) ## [v1.79.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.0) -See changes [here](./CHANGELOG_2022.md#v1790) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1790) ## [v1.78.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.1) -See changes [here](./CHANGELOG_2022.md#v1781) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1781) ## [v1.78.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.0) -See changes [here](./CHANGELOG_2022.md#v1780) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1780) ## [v1.77.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.2) -See changes [here](./CHANGELOG_2022.md#v1772) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1772) ## [v1.77.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.1) -See changes [here](./CHANGELOG_2022.md#v1771) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1771) ## [v1.77.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.0) -See changes [here](./CHANGELOG_2022.md#v1770) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1770) ## [v1.76.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.76.1) -See changes [here](./CHANGELOG_2022.md#v1761) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1761) ## [v1.76.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.76.0) -See changes [here](./CHANGELOG_2022.md#v1760) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1760) ## [v1.75.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.75.1) -See changes [here](./CHANGELOG_2022.md#v1751) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1751) ## [v1.75.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.75.0) -See changes [here](./CHANGELOG_2022.md#v1750) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1750) ## [v1.74.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.74.0) -See changes [here](./CHANGELOG_2022.md#v1740) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1740) ## [v1.73.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.1) -See changes [here](./CHANGELOG_2022.md#v1731) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1731) ## [v1.73.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.0) -See changes [here](./CHANGELOG_2022.md#v1730) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1730) ## [v1.72.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.72.0) -See changes [here](./CHANGELOG_2022.md#v1720) +See changes [here](https://docs.victoriametrics.com/changelog_2022/#v1720) ## [v1.71.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.71.0) -See changes [here](./CHANGELOG_2021.md#v1710) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1710) ## [v1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0) -See changes [here](./CHANGELOG_2021.md#v1700) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1700) ## [v1.69.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.69.0) -See changes [here](./CHANGELOG_2021.md#v1690) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1690) ## [v1.68.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.68.0) -See changes [here](./CHANGELOG_2021.md#v1680) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1680) ## [v1.67.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.67.0) -See changes [here](./CHANGELOG_2021.md#v1670) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1670) ## [v1.66.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.2) -See changes [here](./CHANGELOG_2021.md#v1662) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1662) ## [v1.66.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.1) -See changes [here](./CHANGELOG_2021.md#v1661) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1661) ## [v1.66.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.0) -See changes [here](./CHANGELOG_2021.md#v1660) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1660) ## [v1.65.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.65.0) -See changes [here](./CHANGELOG_2021.md#v1650) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1650) ## [v1.64.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.64.1) -See changes [here](./CHANGELOG_2021.md#v1641) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1641) ## [v1.64.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.64.0) -See changes [here](./CHANGELOG_2021.md#v1640) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1640) ## [v1.63.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.63.0) -See changes [here](./CHANGELOG_2021.md#v1630) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1630) ## [v1.62.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.62.0) -See changes [here](./CHANGELOG_2021.md#v1620) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1620) ## [v1.61.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.1) -See changes [here](./CHANGELOG_2021.md#v1611) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1611) ## [v1.61.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.0) -See changes [here](./CHANGELOG_2021.md#v1610) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1610) ## [v1.60.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.60.0) -See changes [here](./CHANGELOG_2021.md#v1600) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1600) ## [v1.59.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.59.0) -See changes [here](./CHANGELOG_2021.md#v1590) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1590) ## [v1.58.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.58.0) -See changes [here](./CHANGELOG_2021.md#v1580) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1580) ## [v1.57.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.57.1) -See changes [here](./CHANGELOG_2021.md#v1571) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1571) ## [v1.57.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.57.0) -See changes [here](./CHANGELOG_2021.md#v1570) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1570) ## [v1.56.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.56.0) -See changes [here](./CHANGELOG_2021.md#v1560) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1560) ## [v1.55.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.55.1) -See changes [here](./CHANGELOG_2021.md#v1551) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1551) ## [v1.55.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.55.0) -See changes [here](./CHANGELOG_2021.md#v1550) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1550) ## [v1.54.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.54.1) -See changes [here](./CHANGELOG_2021.md#v1541) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1541) ## [v1.54.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.54.0) -See changes [here](./CHANGELOG_2021.md#v1540) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1540) ## [v1.53.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.53.1) -See changes [here](./CHANGELOG_2021.md#v1531) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1531) ## [v1.53.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.53.0) -See changes [here](./CHANGELOG_2021.md#v1530) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1530) ## [v1.52.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.52.0) -See changes [here](./CHANGELOG_2021.md#v1520) +See changes [here](https://docs.victoriametrics.com/changelog_2021/#v1520) ## [v1.51.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.51.0) -See changes [here](./CHANGELOG_2020.md#v1510) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1510) ## [v1.50.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.50.2) -See changes [here](./CHANGELOG_2020.md#v1502) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1502) ## [v1.50.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.50.1) -See changes [here](./CHANGELOG_2020.md#v1501) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1501) ## [v1.50.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.50.0) -See changes [here](./CHANGELOG_2020.md#v1500) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1500) ## [v1.49.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.49.0) -See changes [here](./CHANGELOG_2020.md#v1490) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1490) ## [v1.48.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.48.0) -See changes [here](./CHANGELOG_2020.md#v1480) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1480) ## [v1.47.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.47.0) -See changes [here](./CHANGELOG_2020.md#v1470) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1470) ## [v1.46.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.46.0) -See changes [here](./CHANGELOG_2020.md#v1460) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1460) ## [v1.45.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.45.0) -See changes [here](./CHANGELOG_2020.md#v1450) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1450) ## [v1.44.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.44.0) -See changes [here](./CHANGELOG_2020.md#v1440) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1440) ## [v1.43.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.43.0) -See changes [here](./CHANGELOG_2020.md#v1430) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1430) ## [v1.42.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.42.0) -See changes [here](./CHANGELOG_2020.md#v1420) +See changes [here](https://docs.victoriametrics.com/changelog_2020/#v1420) ## Previous releases diff --git a/docs/CHANGELOG_2020.md b/docs/CHANGELOG_2020.md index d8eda554b..40745663c 100644 --- a/docs/CHANGELOG_2020.md +++ b/docs/CHANGELOG_2020.md @@ -26,7 +26,7 @@ Released at 2020-12-27 Released at 2020-12-19 -* FEATURE: do not publish duplicate Docker images with `-cluster` tag suffix for [vmagent](./vmagent.md), [vmalert](./vmalert.md), [vmauth](./vmauth.md), [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md), since they are identical to images without `-cluster` tag suffix. +* FEATURE: do not publish duplicate Docker images with `-cluster` tag suffix for [vmagent](https://docs.victoriametrics.com/vmagent/), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmauth](https://docs.victoriametrics.com/vmauth/), [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/), since they are identical to images without `-cluster` tag suffix. * BUGFIX: vmalert: properly populate template variables. This has been broken in v1.50.0. See * BUGFIX: properly parse negative combined duration in MetricsQL such as `-1h3m4s`. It must be parsed as `-(1h + 3m + 4s)`. Previously it was parsed as `-1h + 3m + 4s`. @@ -44,7 +44,7 @@ Released at 2020-12-15 Released at 2020-12-15 -* FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described [in these docs](./README.md#backfilling). This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset. +* FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described [in these docs](https://docs.victoriametrics.com/#backfilling). This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset. * FEATURE: vmalert: add `query`, `first` and `value` functions to alert templates. See * FEATURE: vmagent: return user-friendly HTML page when requesting `/targets` page from web browser. The page is returned in the old plaintext format when requesting via curl or similar tool. * FEATURE: allow multiple whitespace chars between measurements, fields and timestamp when parsing InfluxDB line protocol. @@ -71,7 +71,7 @@ Released at 2020-12-15 Released at 2020-12-05 * FEATURE: optimize Consul service discovery speed when discovering big number of services. See -* FEATURE: add `label_uppercase(q, label1, ... labelN)` and `label_lowercase(q, label1, ... labelN)` function to [MetricsQL](./MetricsQL.md) +* FEATURE: add `label_uppercase(q, label1, ... labelN)` and `label_lowercase(q, label1, ... labelN)` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/) for uppercasing and lowercasing values for the given labels. See * FEATURE: add `count_eq_over_time(m[d], N)` and `count_ne_over_time(m[d], N)` for counting the number of samples for `m` over `d` that (equal / not equal) to `N`. * FEATURE: do not print usage info for all the command-line flags when incorrect command-line flag is passed. Previously it could be hard reading the error message @@ -100,7 +100,7 @@ Released at 2020-11-26 * FEATURE: vmselect: add `-replicationFactor` command-line flag for reducing query duration when replication is enabled and a part of vmstorage nodes are temporarily slow and/or temporarily unavailable. See * FEATURE: vminsert: export `vm_rpc_vmstorage_is_reachable` metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes. -* FEATURE: vmagent: add [Netflix Eureka](https://github.com/Netflix/eureka) service discovery (aka [eureka_sd_config](./sd_configs.md#eureka_sd_configs)). See +* FEATURE: vmagent: add [Netflix Eureka](https://github.com/Netflix/eureka) service discovery (aka [eureka_sd_config](https://docs.victoriametrics.com/sd_configs/#eureka_sd_configs)). See * FEATURE: add `filters` option to `dockerswarm_sd_config` like Prometheus did in v2.23.0 - see * FEATURE: expose `__meta_ec2_ipv6_addresses` label for `ec2_sd_config` like Prometheus will do in the next release. * FEATURE: add `-loggerWarnsPerSecondLimit` command-line flag for rate limiting of WARN messages in logs. See @@ -108,7 +108,7 @@ Released at 2020-11-26 at the rate exceeding the given limit. See * FEATURE: add remoteAddr to slow query log in order to simplify identifying the client that sends slow queries to VictoriaMetrics. Slow query logging is controlled with `-search.logSlowQueryDuration` command-line flag. -* FEATURE: add `/tags/delSeries` handler from Graphite Tags API. See <./README.md#graphite-tags-api-usage> +* FEATURE: add `/tags/delSeries` handler from Graphite Tags API. See * FEATURE: log metric name plus all its labels when the metric timestamp is out of the configured retention. This should simplify detecting the source of metrics with unexpected timestamps. * FEATURE: add `-dryRun` command-line flag to single-node VictoriaMetrics in order to check config file pointed by `-promscrape.config`. @@ -138,9 +138,9 @@ Released at 2020-11-16 See * FEATURE: update Go builder from v1.15.4 to v1.15.5. This should fix [these issues in Go](https://github.com/golang/go/issues?q=milestone%3AGo1.15.5+label%3ACherryPickApproved). * FEATURE: added `/internal/force_flush` http handler for flushing recently ingested data from in-memory buffers to persistent storage. - See [troubleshooting docs](./README.md#troubleshooting) for more details. + See [troubleshooting docs](https://docs.victoriametrics.com/#troubleshooting) for more details. * FEATURE: added [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html) support. - See [these docs](./README.md#graphite-tags-api-usage) for details. + See [these docs](https://docs.victoriametrics.com/#graphite-tags-api-usage) for details. * BUGFIX: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range. See and @@ -163,7 +163,7 @@ Released at 2020-11-07 * BUGFIX: vmagent: eliminate data race when `-promscrape.streamParse` command-line is set. Previously this mode could result in scraped metrics with garbage labels. See for details. -* BUGFIX: properly calculate `topk_*` and `bottomk_*` functions from [MetricsQL](./MetricsQL.md) for time series with gaps. +* BUGFIX: properly calculate `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/metricsql/) for time series with gaps. See ## [v1.45.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.45.0) @@ -181,9 +181,9 @@ Released at 2020-11-02 * FEATURE: vmagent: add `stream parse` mode. This mode allows reducing memory usage when individual scrape targets expose tens of millions of metrics. For example, during scraping Prometheus in [federation](https://prometheus.io/docs/prometheus/latest/federation/) mode. See `-promscrape.streamParse` command-line option and `stream_parse: true` config option for `scrape_config` section in `-promscrape.config`. - See and [troubleshooting docs for vmagent](./vmagent.md#troubleshooting). + See and [troubleshooting docs for vmagent](https://docs.victoriametrics.com/vmagent/#troubleshooting). * FEATURE: vmalert: add `-dryRun` command-line option for validating the provided config files without the need to start `vmalert` service. -* FEATURE: accept optional third argument of string type at `topk_*` and `bottomk_*` functions. This is label name for additional time series to return with the sum of time series outside top/bottom K. See [MetricsQL docs](./MetricsQL.md) for more details. +* FEATURE: accept optional third argument of string type at `topk_*` and `bottomk_*` functions. This is label name for additional time series to return with the sum of time series outside top/bottom K. See [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for more details. * FEATURE: vmagent: expose `/api/v1/targets` page according to [the corresponding Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/#targets). See @@ -213,11 +213,11 @@ Released at 2020-10-13 node_filesystem_files{ host="$host", mountpoint="/" } - node_filesystem_files_free ``` -* FEATURE: vmagent: add Docker Swarm service discovery (aka [dockerswarm_sd_config](./sd_configs.md#dockerswarm_sd_configs)). +* FEATURE: vmagent: add Docker Swarm service discovery (aka [dockerswarm_sd_config](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs)). See -* FEATURE: add ability to export data in CSV format. See [these docs](./README.md#how-to-export-csv-data) for details. +* FEATURE: add ability to export data in CSV format. See [these docs](https://docs.victoriametrics.com/#how-to-export-csv-data) for details. * FEATURE: vmagent: add `-promscrape.suppressDuplicateScrapeTargetErrors` command-line flag for suppressing `duplicate scrape target` errors. - See and <./vmagent.md#troubleshooting> . + See and . * FEATURE: vmagent: show original labels before relabeling is applied on `duplicate scrape target` errors. This should simplify debugging for incorrect relabeling. See * FEATURE: vmagent: `/targets` page now accepts optional `show_original_labels=1` query arg for displaying original labels for each target before relabeling is applied. @@ -226,9 +226,9 @@ Released at 2020-10-13 The final merge is started after no new data is ingested into per-month partition during `-finalMergeDelay`. * FEATURE: add `vm_rows_added_to_storage_total` metric, which shows the total number of rows added to storage since app start. The `sum(rate(vm_rows_added_to_storage_total))` can be smaller than `sum(rate(vm_rows_inserted_total))` if certain metrics are dropped - due to [relabeling](./README.md#relabeling). The `sum(rate(vm_rows_added_to_storage_total))` can be bigger - than `sum(rate(vm_rows_inserted_total))` if [replication](./Cluster-VictoriaMetrics.md#replication-and-data-safety) is enabled. -* FEATURE: keep metric name after applying [MetricsQL](./MetricsQL.md) functions, which don't change time series meaning. + due to [relabeling](https://docs.victoriametrics.com/#relabeling). The `sum(rate(vm_rows_added_to_storage_total))` can be bigger + than `sum(rate(vm_rows_inserted_total))` if [replication](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) is enabled. +* FEATURE: keep metric name after applying [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions, which don't change time series meaning. The list of such functions: * `keep_last_value` * `keep_next_value` @@ -269,8 +269,8 @@ Released at 2020-10-13 Released at 2020-10-06 * FEATURE: reduce CPU usage for repeated queries over sliding time window when no new time series are added to the database. - Typical use cases: repeated evaluation of alerting rules in [vmalert](./vmalert.md) or dashboard auto-refresh in Grafana. -* FEATURE: vmagent: add OpenStack service discovery aka [openstack_sd_config](./sd_configs.md#openstack_sd_configs). + Typical use cases: repeated evaluation of alerting rules in [vmalert](https://docs.victoriametrics.com/vmalert/) or dashboard auto-refresh in Grafana. +* FEATURE: vmagent: add OpenStack service discovery aka [openstack_sd_config](https://docs.victoriametrics.com/sd_configs/#openstack_sd_configs). See . * FEATURE: vmalert: make `-maxIdleConnections` configurable for datasource HTTP client. This option can be used for minimizing connection churn. See . @@ -289,10 +289,10 @@ Released at 2020-10-06 Released at 2020-09-30 * FEATURE: use all the available CPU cores when accepting data via a single TCP connection - for [all the supported protocols](./README.md#how-to-import-time-series-data). + for [all the supported protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data). Previously data ingested via a single TCP connection could use only a single CPU core. This could limit data ingestion performance. The main benefit of this feature is that data can be imported at max speed via a single connection - there is no need to open multiple concurrent - connections to VictoriaMetrics or [vmagent](./vmagent.md) in order to achieve the maximum data ingestion speed. + connections to VictoriaMetrics or [vmagent](https://docs.victoriametrics.com/vmagent/) in order to achieve the maximum data ingestion speed. * FEATURE: cluster: improve performance for data ingestion path from `vminsert` to `vmstorage` nodes. The maximum data ingestion performance for a single connection between `vminsert` and `vmstorage` node scales with the number of available CPU cores on `vmstorage` side. This should help with . @@ -300,7 +300,7 @@ Released at 2020-09-30 This is the most optimized approach for data migration between VictoriaMetrics instances. Both single-node and cluster instances are supported. See . * FEATURE: add `reduce_mem_usage` query option to `/api/v1/export` in order to reduce memory usage during data export / import. - See [these docs](./README.md#how-to-export-data-in-json-line-format) for details. + See [these docs](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format) for details. * FEATURE: improve performance for `/api/v1/series` handler when it returns big number of time series. * FEATURE: add `vm_merge_need_free_disk_space` metric, which can be used for estimating the number of deferred background data merges due to the lack of free disk space. See . @@ -312,7 +312,7 @@ Released at 2020-09-30 The adjustment was increasing the resulting GOMAXPROCS by 1, which looked confusing to users. See . * BUGFIX: vmagent: do not show `-remoteWrite.url` in initial logs if `-remoteWrite.showURL` isn't set. See . -* BUGFIX: properly handle case when [/metrics/find](./README.md#graphite-metrics-api-usage) finds both a leaf and a node for the given `query=prefix.*`. +* BUGFIX: properly handle case when [/metrics/find](https://docs.victoriametrics.com/#graphite-metrics-api-usage) finds both a leaf and a node for the given `query=prefix.*`. In this case only the node must be returned with stripped dot in the end of id as carbonapi does. ## Previous releases diff --git a/docs/CHANGELOG_2021.md b/docs/CHANGELOG_2021.md index b152a920c..9f5f41187 100644 --- a/docs/CHANGELOG_2021.md +++ b/docs/CHANGELOG_2021.md @@ -20,76 +20,76 @@ into increase of CPU usage until merges are finished. We recommend updating in "off-peak" time when load on the VictoriaMetrics is on its minimum. -* FEATURE: [VictoriaMetrics enterprise](./enterprise.md): add multi-level downsampling support. See [these docs](./README.md#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36). -* FEATURE: [vmui](./README.md#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916). -* FEATURE: [vmui](./README.md#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging. -* FEATURE: accept optional `extra_filters[]=series_selector` query args at Prometheus query APIs additionally to `extra_label` query args. This allows enforcing additional filters for all the Prometheus query APIs by using [vmgateway](./vmgateway.md) or [vmauth](./vmauth.md). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863). -* FEATURE: [vmauth](./vmauth.md): allow specifying `http` and `https` urls in `-auth.config` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1898). Thanks for @TFM93 . -* FEATURE: [vmagent](./vmagent.md): allow specifying `http` and `https` urls in the following command-line flags: `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`. +* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging. +* FEATURE: accept optional `extra_filters[]=series_selector` query args at Prometheus query APIs additionally to `extra_label` query args. This allows enforcing additional filters for all the Prometheus query APIs by using [vmgateway](https://docs.victoriametrics.com/vmgateway/) or [vmauth](https://docs.victoriametrics.com/vmauth/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow specifying `http` and `https` urls in `-auth.config` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1898). Thanks for @TFM93 . +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying `http` and `https` urls in the following command-line flags: `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`. * FEATURE: vminsert: allow specifying `http` and `https` urls in `-relabelConfig` command-line flag. * FEATURE: vminsert: add `-maxLabelValueLen` command-line flag for the ability to configure the maximum length of label value. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908). -* FEATURE: preserve the order of time series passed to [limit_offset](./MetricsQL.md#limit_offset) function. This allows implementing series paging via `limit_offset(limit, offset, sort_by_label(...))`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951) issues. -* FEATURE: automatically convert `(value1|...|valueN)` into `{value1,...,valueN}` inside `__graphite__` pseudo-label. This allows using [Grafana multi-value template variables](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, `{__graphite__=~"foo.($bar)"}` is expanded to `{__graphite__=~"foo.{x,y}"}` if both `x` and `y` are selected for `$bar` template variable. See [these docs](./README.md#selecting-graphite-metrics) for details. -* FEATURE: add [timestamp_with_name](./MetricsQL.md#timestamp_with_name) function. It works the same as [timestamp](./MetricsQL.md#timestamp), but leaves the original time series names, so it can be used in queries, which match multiple time series names: `timestamp_with_name({foo="bar"}[1h])`. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388) for more context. -* FEATURE: add [changes_prometheus](./MetricsQL.md#changes_prometheus), [increase_prometheus](./MetricsQL.md#increase_prometheus) and [delta_prometheus](./MetricsQL.md#delta_prometheus) functions, which don't take into account the previous sample before the given lookbehind window specified in square brackets. These functions may be used when the Prometheus behaviour for `changes()`, `increase()` and `delta()` functions is needed to be preserved. VictoriaMetrics uses slightly different behaviour for `changes()`, `increase()` and `delta()` functions by default - see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1962). +* FEATURE: preserve the order of time series passed to [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) function. This allows implementing series paging via `limit_offset(limit, offset, sort_by_label(...))`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951) issues. +* FEATURE: automatically convert `(value1|...|valueN)` into `{value1,...,valueN}` inside `__graphite__` pseudo-label. This allows using [Grafana multi-value template variables](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, `{__graphite__=~"foo.($bar)"}` is expanded to `{__graphite__=~"foo.{x,y}"}` if both `x` and `y` are selected for `$bar` template variable. See [these docs](https://docs.victoriametrics.com/#selecting-graphite-metrics) for details. +* FEATURE: add [timestamp_with_name](https://docs.victoriametrics.com/metricsql/#timestamp_with_name) function. It works the same as [timestamp](https://docs.victoriametrics.com/metricsql/#timestamp), but leaves the original time series names, so it can be used in queries, which match multiple time series names: `timestamp_with_name({foo="bar"}[1h])`. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388) for more context. +* FEATURE: add [changes_prometheus](https://docs.victoriametrics.com/metricsql/#changes_prometheus), [increase_prometheus](https://docs.victoriametrics.com/metricsql/#increase_prometheus) and [delta_prometheus](https://docs.victoriametrics.com/metricsql/#delta_prometheus) functions, which don't take into account the previous sample before the given lookbehind window specified in square brackets. These functions may be used when the Prometheus behaviour for `changes()`, `increase()` and `delta()` functions is needed to be preserved. VictoriaMetrics uses slightly different behaviour for `changes()`, `increase()` and `delta()` functions by default - see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1962). * BUGFIX: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures, which has been introduced in v1.70.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1944). -* BUGFIX: [vmalert](./vmalert.md): restore the ability to use `$labels.alertname` in labels templating. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921). -* BUGFIX: [vmui](./README.md#vmui): add missing `query` caption to the input field for the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1900). -* BUGFIX: [vmui](./README.md#vmui): fix navigation over query history with `Ctrl+up/down` and fix zoom relatively to the cursor position. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1936). -* BUGFIX: deduplicate samples more thoroughly if [deduplication](./README.md#deduplication) is enabled. Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in bigger storage space requirements. -* BUGFIX: [vmagent](./vmagent.md): follow up to 5 redirects when `follow_redirects: true` is set for a particular scrape config. Previously only a single redirect was performed in this case. It is expected these redirects are performed to the original hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945). -* BUGFIX: de-duplicate data exported via [/api/v1/export/csv](./README.md#how-to-export-csv-data) by default if [deduplication](./README.md#deduplication) is enabled. The de-duplication can be disabled by passing `reduce_mem_usage=1` query arg to `/api/v1/export/csv`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837). -* BUGFIX: [vmalert](./vmalert.md): properly store [historical data](./vmalert.md#rules-backfilling) to old Prometheus versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): restore the ability to use `$labels.alertname` in labels templating. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): add missing `query` caption to the input field for the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1900). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix navigation over query history with `Ctrl+up/down` and fix zoom relatively to the cursor position. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1936). +* BUGFIX: deduplicate samples more thoroughly if [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in bigger storage space requirements. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): follow up to 5 redirects when `follow_redirects: true` is set for a particular scrape config. Previously only a single redirect was performed in this case. It is expected these redirects are performed to the original hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945). +* BUGFIX: de-duplicate data exported via [/api/v1/export/csv](https://docs.victoriametrics.com/#how-to-export-csv-data) by default if [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. The de-duplication can be disabled by passing `reduce_mem_usage=1` query arg to `/api/v1/export/csv`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly store [historical data](https://docs.victoriametrics.com/vmalert/#rules-backfilling) to old Prometheus versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943). ## [v1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0) Released at 2021-12-02 -* FEATURE: [vmalert](./vmalert.md): add ability to pass arbitrary query args to `-datasource.url` on a per-group basis via `params` option. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1892). -* FEATURE: add `now()` function to MetricsQL. This function returns the current timestamp in seconds. See [these docs](./MetricsQL.md#now). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to pass arbitrary query args to `-datasource.url` on a per-group basis via `params` option. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1892). +* FEATURE: add `now()` function to MetricsQL. This function returns the current timestamp in seconds. See [these docs](https://docs.victoriametrics.com/metricsql/#now). * FEATURE: vmauth: allow using optional `name` field in configs. This field is then used as `username` label value for `vmauth_user_requests_total` metric. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1805). * FEATURE: vmagent: export `vm_persistentqueue_read_duration_seconds_total` and `vm_persistentqueue_write_duration_seconds_total` metrics, which can be used for detecting persistent queue saturation with `rate(vm_persistentqueue_write_duration_seconds_total) > 0.9` alerting rule. * FEATURE: export `vm_filestream_read_duration_seconds_total` and `vm_filestream_write_duration_seconds_total` metrics, which can be used for detecting persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9` alerting rule. * FEATURE: export `vm_cache_size_max_bytes` metrics, which show capacity for various caches. These metrics can be used for determining caches with reach its capacity with `vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9` query. -* FEATURE: [vmbackup](./vmbackup.md), [vmrestore](./vmrestore.md): add `-s3ForcePathStyle` command-line flag, which can be used for making backups to [Aliyun OSS](https://www.aliyun.com/product/oss). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1802). -* FEATURE: [vmctl](./vmctl.md): improve data migration from OpenTSDB. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1809). Thanks to @johnseekins . +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/): add `-s3ForcePathStyle` command-line flag, which can be used for making backups to [Aliyun OSS](https://www.aliyun.com/product/oss). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1802). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): improve data migration from OpenTSDB. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1809). Thanks to @johnseekins . * FEATURE: suppress `connection reset by peer` errors when remote client resets TCP connection to VictoriaMetrics / vmagent while ingesting the data via InfluxDB line protocol, Graphite protocol or OpenTSDB protocol. This error is expected, so there is no need in logging it. -* FEATURE: [vmui](./README.md#vmui): store the display type in URL, so it isn't lost when copy-pasting the URL. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1804). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): store the display type in URL, so it isn't lost when copy-pasting the URL. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1804). * FEATURE: vmalert: make `-notifier.url` command-line flag optional. This flag can be omitted if `vmalert` is used solely for recording rules and doesn't evaluate alerting rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1870). -* FEATURE: [vmbackup](./vmbackup.md), [vmrestore](./vmrestore.md): export internal metrics at `http://vmbackup:8420/metrics` and `http://vmrestore:8421/metrics` for better visibility of the backup/restore process. -* FEATURE: allow trailing whitespace after the timestamp when [parsing Graphite plaintext lines](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/): export internal metrics at `http://vmbackup:8420/metrics` and `http://vmrestore:8421/metrics` for better visibility of the backup/restore process. +* FEATURE: allow trailing whitespace after the timestamp when [parsing Graphite plaintext lines](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865). * FEATURE: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does. This is needed for improving integration with third-party solutions, which rely on these endpoints. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833). * BUGFIX: vmagent: prevent from scraping duplicate targets if `-promscrape.dropOriginalLabels` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1830). Thanks to @guidao for the fix. -* BUGFIX: vmstorage [enterprise](./enterprise.md): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](./PerTenantStatistic.md) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605). +* BUGFIX: vmstorage [enterprise](https://docs.victoriametrics.com/enterprise/): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/pertenantstatistic/) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605). * BUGFIX: vmauth: properly take into account the value passed to `-maxIdleConnsPerBackend` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300). -* BUGFIX: vmagent: fix [reading data from Kafka](./vmagent.md#reading-metrics-from-kafka). -* BUGFIX: vmalert: fix [replay mode](./vmalert.md#rules-backfilling) in enterprise version. -* BUGFIX: consistently return zero from [deriv()](./MetricsQL.md#deriv) function applied to a constant time series. Previously it could return small non-zero values in this case. -* BUGFIX: [vmrestore](./vmrestore.md): properly resume downloading for partially downloaded big files. Previously such files were re-downloaded from the beginning after the interrupt. Now only the remaining parts of the file are downloaded. This allows saving network bandwidth. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487). -* BUGFIX: [vmui](./README.md#vmui): do not store the last query across vmui page reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1694). -* BUGFIX: [vmui](./README.md#vmui): fix `Cannot read properties of undefined` error at table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1797). +* BUGFIX: vmagent: fix [reading data from Kafka](https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka). +* BUGFIX: vmalert: fix [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling) in enterprise version. +* BUGFIX: consistently return zero from [deriv()](https://docs.victoriametrics.com/metricsql/#deriv) function applied to a constant time series. Previously it could return small non-zero values in this case. +* BUGFIX: [vmrestore](https://docs.victoriametrics.com/vmrestore/): properly resume downloading for partially downloaded big files. Previously such files were re-downloaded from the beginning after the interrupt. Now only the remaining parts of the file are downloaded. This allows saving network bandwidth. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not store the last query across vmui page reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1694). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `Cannot read properties of undefined` error at table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1797). ## [v1.69.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.69.0) Released at 2021-11-08 * FEATURE: vmalert: allow groups with empty rules list like Prometheus does. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1742). -* FEATURE: vmalert: allow groups with default `tenant` in `-clusterMode`. Default `tenant` values can be specified via `-defaultTenant.prometheus` and `-defaultTenant.graphite`. See [these docs](./vmalert.md#multitenancy). +* FEATURE: vmalert: allow groups with default `tenant` in `-clusterMode`. Default `tenant` values can be specified via `-defaultTenant.prometheus` and `-defaultTenant.graphite`. See [these docs](https://docs.victoriametrics.com/vmalert/#multitenancy). * FEATURE: vmagent: add `collapse` and `expand` buttons per each group of targets with the same `job_name` at `http://vmagent:8429/targets` page. -* FEATURE: automatically detect timestamp precision (ns, us, ms or s) for the data ingested into VictoriaMetrics via [InfluxDB line protocol](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). +* FEATURE: automatically detect timestamp precision (ns, us, ms or s) for the data ingested into VictoriaMetrics via [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). * FEATURE: vmagent: add ability to protect `/config` page with auth key via `-configAuthKey` command-line flag. This page may contain sensitive config information, so it may be good to restrict access to this page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764). * FEATURE: vmagent: hide passwords and auth tokens at `/config` page like Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764). * FEATURE: vmagent: add `-promscrape.maxResponseHeadersSize` command-line flag for tuning the maximum HTTP response headers size for Prometheus scrape targets. * FEATURE: vmagent: send data to multiple configured remote storage systems in parallel (e.g. when multiple `-remoteWrite.url` flag values are specified). This should improve data ingestion speed. * FEATURE: vmagent: add `-remoteWrite.maxRowsPerBlock` command-line flag for tuning the number of samples to send to remote storage per each block. Bigger values may improve data ingestion performance at the cost of higher memory usage. -* FEATURE: vmagent: distribute Kafka messages among all the partitions when [writing data to Kafka](./vmagent.md#writing-metrics-to-kafka). -* FEATURE: add [label_graphite_group](./MetricsQL.md#label_graphite_group) function for extracting the given groups from Graphite metric names. -* FEATURE: add [duration_over_time](./MetricsQL.md#duration_over_time) function for calculating the actual lifetime of the time series with possible gaps. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780). -* FEATURE: add [limit_offset](./MetricsQL.md#limit_offset) function, which can be used for implementing simple paging over big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778). +* FEATURE: vmagent: distribute Kafka messages among all the partitions when [writing data to Kafka](https://docs.victoriametrics.com/vmagent/#writing-metrics-to-kafka). +* FEATURE: add [label_graphite_group](https://docs.victoriametrics.com/metricsql/#label_graphite_group) function for extracting the given groups from Graphite metric names. +* FEATURE: add [duration_over_time](https://docs.victoriametrics.com/metricsql/#duration_over_time) function for calculating the actual lifetime of the time series with possible gaps. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780). +* FEATURE: add [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) function, which can be used for implementing simple paging over big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778). -* BUGFIX: vmagent: reduce the increased memory usage when scraping targets with big number of metrics which periodically change. The memory usage has been increased in v1.68.0 after vmagent started generating staleness markers in [stream parse mode](./vmagent.md#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745). +* BUGFIX: vmagent: reduce the increased memory usage when scraping targets with big number of metrics which periodically change. The memory usage has been increased in v1.68.0 after vmagent started generating staleness markers in [stream parse mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745). * BUGFIX: vmagent: properly display `proxy_url` config option at `http://vmagent:8429/config` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755). * BUGFIX: fix tests for Apple M1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653). @@ -100,43 +100,43 @@ Released at 2021-10-22 * FEATURE: vmagent: expose `-promscrape.config` contents at `/config` page as Prometheus does. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695). * FEATURE: vmagent: add `show original labels` button per each scrape target displayed at `http://vmagent:8429/targets` page. This should improve debuggability for service discovery and relabeling issues similar to [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1664). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698). * FEATURE: vmagent: shard targets among cluster nodes after the relabeling is applied. This should guarantee that targets with the same set of labels go to the same `vmagent` node in the cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687). -* FEATURE: vmagent: automatically switch to [stream parsing mode](./vmagent.md#stream-parsing-mode) if the response from the given target exceeds the command-line flag value `-promscrape.minResponseSizeForStreamParse`. This should reduce memory usage when `vmagent` scrapes targets with non-uniform response sizes (this is the case in Kubernetes monitoring). -* FEATURE: vmagent: send Prometheus-like staleness marks in [stream parsing mode](./vmagent.md#stream-parsing-mode). Previously staleness marks weren't sent in stream parsing mode. See [these docs](./vmagent.md#prometheus-staleness-markers) for details. -* FEATURE: vmagent: properly calculate `scrape_series_added` metric for targets in [stream parsing mode](./vmagent.md#stream-parsing-mode). Previously it was set to 0 in stream parsing mode. See [more details about this metric](https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series). -* FEATURE: vmagent: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics at `http://vmagent:8429/metrics` for scrape targets with the [enabled series limiter](./vmagent.md#cardinality-limiter). -* FEATURE: vmagent: return error if `sample_limit` or `series_limit` options are set when [stream parsing mode](./vmagent.md#stream-parsing-mode) is enabled, since these limits cannot be applied in stream parsing mode. +* FEATURE: vmagent: automatically switch to [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) if the response from the given target exceeds the command-line flag value `-promscrape.minResponseSizeForStreamParse`. This should reduce memory usage when `vmagent` scrapes targets with non-uniform response sizes (this is the case in Kubernetes monitoring). +* FEATURE: vmagent: send Prometheus-like staleness marks in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously staleness marks weren't sent in stream parsing mode. See [these docs](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for details. +* FEATURE: vmagent: properly calculate `scrape_series_added` metric for targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously it was set to 0 in stream parsing mode. See [more details about this metric](https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series). +* FEATURE: vmagent: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics at `http://vmagent:8429/metrics` for scrape targets with the [enabled series limiter](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). +* FEATURE: vmagent: return error if `sample_limit` or `series_limit` options are set when [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is enabled, since these limits cannot be applied in stream parsing mode. * FEATURE: vmalert: add `-remoteRead.disablePathAppend` command-line flag, which allows specifying the full `-remoteRead.url`. If `-remoteRead.disablePathAppend` is set, then `vmalert` doesn't add `/api/v1/query` suffix to `-remoteRead.url`. -* FEATURE: add trigonometric functions, which are going to be added in [Prometheus 2.31](https://github.com/prometheus/prometheus/pull/9239): [acosh](./MetricsQL.md#acosh), [asinh](./MetricsQL.md#asinh), [atan](./MetricsQL.md#atan), [atanh](./MetricsQL.md#atanh), [cosh](./MetricsQL.md#cosh), [deg](./MetricsQL.md#deg), [rad](./MetricsQL.md#rad), [sinh](./MetricsQL.md#sinh), [tan](./MetricsQL.md#tan), [tanh](./MetricsQL.md#tanh). Also add `atan2` binary operator. See [this pull request](https://github.com/prometheus/prometheus/pull/9248). -* FEATURE: consistently return the same set of time series from [limitk](./MetricsQL.md#limitk) function. This improves the usability of periodically refreshed graphs. -* FEATURE: [vmui](./README.md#vmui): varios UX improvements. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1711) and [these docs](./README.md#vmui). -* FEATURE: [vmauth](./vmauth.md): add ability to specify HTTP headers, which will be sent in requests to backends. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1736). +* FEATURE: add trigonometric functions, which are going to be added in [Prometheus 2.31](https://github.com/prometheus/prometheus/pull/9239): [acosh](https://docs.victoriametrics.com/metricsql/#acosh), [asinh](https://docs.victoriametrics.com/metricsql/#asinh), [atan](https://docs.victoriametrics.com/metricsql/#atan), [atanh](https://docs.victoriametrics.com/metricsql/#atanh), [cosh](https://docs.victoriametrics.com/metricsql/#cosh), [deg](https://docs.victoriametrics.com/metricsql/#deg), [rad](https://docs.victoriametrics.com/metricsql/#rad), [sinh](https://docs.victoriametrics.com/metricsql/#sinh), [tan](https://docs.victoriametrics.com/metricsql/#tan), [tanh](https://docs.victoriametrics.com/metricsql/#tanh). Also add `atan2` binary operator. See [this pull request](https://github.com/prometheus/prometheus/pull/9248). +* FEATURE: consistently return the same set of time series from [limitk](https://docs.victoriametrics.com/metricsql/#limitk) function. This improves the usability of periodically refreshed graphs. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): varios UX improvements. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1711) and [these docs](https://docs.victoriametrics.com/#vmui). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to specify HTTP headers, which will be sent in requests to backends. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1736). * FEATURE: add `/flags` page to all the VictoriaMetrics components. This page contains command-line flags passed to the component. -* FEATURE: allow using tab separators additionally to whitespace separators when [ingesting data in Graphite plaintext protocol](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). Such separators are [supported by Carbon-c-relay](https://github.com/grobian/carbon-c-relay/commit/f3ffe6cc2b52b07d14acbda649ad3fd6babdd528). +* FEATURE: allow using tab separators additionally to whitespace separators when [ingesting data in Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). Such separators are [supported by Carbon-c-relay](https://github.com/grobian/carbon-c-relay/commit/f3ffe6cc2b52b07d14acbda649ad3fd6babdd528). * BUGFIX: vmstorage: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures (arm and 386). The panic has been introduced in v1.67.0. * BUGFIX: vmalert, vmauth: prevent from frequent closing of TCP connections established to backends under high load. This should reduce the number of TCP sockets in `TIME_WAIT` state at `vmalert` and `vmauth` under high load. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1704). * BUGFIX: vmalert: correctly calculate alert ID including extra labels. Previously, ID for alert entity was generated without alertname or groupname. This led to collision, when multiple alerting rules within the same group producing same labelsets. E.g. expr: `sum(metric1) by (job) > 0` and expr: `sum(metric2) by (job) > 0` could result into same labelset `job: "job"`. The bugfix adds all extra labels right after receiving response from the datasource. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1734). -* BUGFIX: vmalert: fix links in [Web UI](./vmalert.md#web). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1717). -* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](./sd_configs.md#scrape_configs) if this options isn't set explicitly. This aligns the behaviour with Prometheus. +* BUGFIX: vmalert: fix links in [Web UI](https://docs.victoriametrics.com/vmalert/#web). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1717). +* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) if this options isn't set explicitly. This aligns the behaviour with Prometheus. * BUGFIX: vmagent: group scrape targets by the original job names at `http://vmagent:8429/targets` page like Prometheus does. Previously they were grouped by the job name after relabeling, which may result in unexpected empty target groups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707). -* BUGFIX: [vmctl](./vmctl.md): fix importing boolean fields from InfluxDB line protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1709). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix importing boolean fields from InfluxDB line protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1709). ## [v1.67.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.67.0) Released at 2021-10-08 -* FEATURE: add ability to accept metrics from [DataDog agent](https://docs.datadoghq.com/agent/) and [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent). This option simplifies the migration path from DataDog to VictoriaMetrics. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206). -* FEATURE: vmagent [enterprise](./enterprise.md): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](./vmagent.md#kafka-integration). -* FEATURE: vmui: switch to [μPlot](https://github.com/leeoniya/uPlot) and add ability to naturally scroll and zoom graphs. See [these docs](./README.md#vmui). Thanks to @Loori-R. +* FEATURE: add ability to accept metrics from [DataDog agent](https://docs.datadoghq.com/agent/) and [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-datadog-agent). This option simplifies the migration path from DataDog to VictoriaMetrics. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206). +* FEATURE: vmagent [enterprise](https://docs.victoriametrics.com/enterprise/): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](https://docs.victoriametrics.com/vmagent/#kafka-integration). +* FEATURE: vmui: switch to [μPlot](https://github.com/leeoniya/uPlot) and add ability to naturally scroll and zoom graphs. See [these docs](https://docs.victoriametrics.com/#vmui). Thanks to @Loori-R. * FEATURE: vmstorage: stop accepting new data if `-storageDataPath` directory contains less than `-storage.minFreeDiskSpaceBytes` of free space. This should prevent from `out of disk space` crashes. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269). -* FEATURE: calculate quantiles in the same way as Prometheus does in such functions as [quantile_over_time](./MetricsQL.md#quantile_over_time) and [quantile](./MetricsQL.md#quantile). Previously results from VictoriaMetrics could be slightly different than results from Prometheus. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612) issues. -* FEATURE: add `rollup_scrape_interval(m[d])` function to [MetricsQL](./MetricsQL.md), which returns `min`, `max` and `avg` values for the interval between samples for `m` on the given lookbehind window `d`. -* FEATURE: add `topk_last(k, q)` and `bottomk_last(k, q)` functions to [MetricsQL](./MetricsQL.md), which return up to `k` time series from `q` with the maximum / minimum last value on the graph. +* FEATURE: calculate quantiles in the same way as Prometheus does in such functions as [quantile_over_time](https://docs.victoriametrics.com/metricsql/#quantile_over_time) and [quantile](https://docs.victoriametrics.com/metricsql/#quantile). Previously results from VictoriaMetrics could be slightly different than results from Prometheus. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612) issues. +* FEATURE: add `rollup_scrape_interval(m[d])` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which returns `min`, `max` and `avg` values for the interval between samples for `m` on the given lookbehind window `d`. +* FEATURE: add `topk_last(k, q)` and `bottomk_last(k, q)` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which return up to `k` time series from `q` with the maximum / minimum last value on the graph. * BUGFIX: align behavior of the queries `a or on (labels) b`, `a and on (labels) b` and `a unless on (labels) b` where `b` has multiple time series with the given `labels` to Prometheus behavior. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643). * BUGFIX: vmagent: fix `openstack_sd_config` service discovery when both `domain_name` and `project_id` config options are set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655). -* BUGFIX: return proper values (zeroes) from [stddev_over_time](./MetricsQL.md#stddev_over_time) and [stdvar_over_time](./MetricsQL.md#stdvar_over_time) functions when the lookbehind window in square brackets contains only a single sample. Previously the sample value was incorrectly returned in this case. -* BUGFIX: vminsert: fix uneven distribution of time series among storage nodes in [multi-level cluster setup](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672). +* BUGFIX: return proper values (zeroes) from [stddev_over_time](https://docs.victoriametrics.com/metricsql/#stddev_over_time) and [stdvar_over_time](https://docs.victoriametrics.com/metricsql/#stdvar_over_time) functions when the lookbehind window in square brackets contains only a single sample. Previously the sample value was incorrectly returned in this case. +* BUGFIX: vminsert: fix uneven distribution of time series among storage nodes in [multi-level cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672). ## [v1.66.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.2) @@ -153,7 +153,7 @@ Released at 2021-09-22 * FEATURE: add `-cluster` and/or `-enterprise` suffixes to `short_version` label at `vm_app_version` metric exposed at `/metrics` page of every VictoriaMetrics component. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635). -* BUGFIX: vmselect: fix accessing [Graphite APIs](./README.md#graphite-api-usage). The access has been broken in v1.66.0, because `/graphite/*` path prefix accidentally clashed with `/graph*` path prefix used for VictoriaMetrics UI (aka `vmui`). +* BUGFIX: vmselect: fix accessing [Graphite APIs](https://docs.victoriametrics.com/#graphite-api-usage). The access has been broken in v1.66.0, because `/graphite/*` path prefix accidentally clashed with `/graph*` path prefix used for VictoriaMetrics UI (aka `vmui`). * BUGFIX: fix parsing `regex: ` in relabeling rules (for example, `regex: true` or `regex: 123`). The bug has been introduced in v1.66.0. ## [v1.66.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.0) @@ -163,65 +163,65 @@ Released at 2021-09-20 * FEATURE: vmalert: add web UI with the list of alerting groups, alerts and alert statuses. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1602). * FEATURE: vmalert: add `-rule.maxResolveDuration` command-line flag, which could be used for limiting the auto-resolve duration for the alerting rule. By default it is limited to 3x evaluation interval. This could be too high for big evaluation intervals. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1586). * FEATURE: vmalert: add support for Bearer token authorization for `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1608). -* FEATURE: vmagent: send stale markers for disappeared metrics like Prometheus does. Previously stale markers were sent only when the scrape target disappears or when it becomes temporarily unavailable. See [these docs](./vmagent.md#prometheus-staleness-markers) for details. -* FEATURE: vmagent: add ability to set `series_limit` option for a particular scrape target via `__series_limit__` label. This allows setting the limit on the number of time series on a per-target basis. See [these docs](./vmagent.md#cardinality-limiter) for details. -* FEATURE: vmagent: add ability to set `stream_parse` option for a particular scrape target via `__stream_parse__` label. This allows managing the stream parsing mode on a per-target basis. See [these docs](./vmagent.md#stream-parsing-mode) for details. +* FEATURE: vmagent: send stale markers for disappeared metrics like Prometheus does. Previously stale markers were sent only when the scrape target disappears or when it becomes temporarily unavailable. See [these docs](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for details. +* FEATURE: vmagent: add ability to set `series_limit` option for a particular scrape target via `__series_limit__` label. This allows setting the limit on the number of time series on a per-target basis. See [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) for details. +* FEATURE: vmagent: add ability to set `stream_parse` option for a particular scrape target via `__stream_parse__` label. This allows managing the stream parsing mode on a per-target basis. See [these docs](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) for details. * FEATURE: vmagent: add ability to set `scrape_interval` and `scrape_timeout` options for a particular target via `__scrape_interval__` and `__scrape_timeout__` labels in the same way as Prometheus 2.30 does. See [this pull request](https://github.com/prometheus/prometheus/pull/8911). * FEATURE: vmagent: generate `scrape_timeout_seconds` metric per each scrape target, so the target saturation could be calculated with `scrape_duration_seconds / scrape_timeout_seconds`. See the corresponding [pull request from Prometheus 2.30](https://github.com/prometheus/prometheus/pull/9247). * FEATURE: vmagent: reduce CPU usage when calculating the number of newly added series per scrape (this number is sent to remote storage in `scrape_series_added` metric). -* FEATURE: vmagent: reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. See more information about `series_limit` [here](./vmagent.md#cardinality-limiter). -* FEATURE: vminsert: disable rerouting by default when a few of `vmstorage` nodes start accepting data at lower speed than the rest of `vmstorage` nodes. This should improve VictoriaMetrics cluster stability during rolling restarts and during spikes in [time series churn rate](./FAQ.md#what-is-high-churn-rate). The rerouting can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert`. +* FEATURE: vmagent: reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. See more information about `series_limit` [here](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). +* FEATURE: vminsert: disable rerouting by default when a few of `vmstorage` nodes start accepting data at lower speed than the rest of `vmstorage` nodes. This should improve VictoriaMetrics cluster stability during rolling restarts and during spikes in [time series churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). The rerouting can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert`. * FEATURE: vmauth: do not put invalid auth tokens into log by default due to security reasons. The logging can be returned back by passing `-logInvalidAuthTokens` command-line flag to `vmauth`. Requests with invalid auth tokens are counted at `vmagent_http_request_errors_total{reason="invalid_auth_token"}` metric exposed by `vmauth` at `/metrics` page. -* FEATURE: add new relabeling actions: `keep_metrics` and `drop_metrics`. This simplifies metrics filtering by metric names. See [these docs](./vmagent.md#relabeling) for more details. -* FEATURE: allow splitting long `regex` in relabeling filters into an array of shorter regexps, which can be put into multiple lines for better readability and maintainability. See [these docs](./vmagent.md#relabeling) for more details. +* FEATURE: add new relabeling actions: `keep_metrics` and `drop_metrics`. This simplifies metrics filtering by metric names. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details. +* FEATURE: allow splitting long `regex` in relabeling filters into an array of shorter regexps, which can be put into multiple lines for better readability and maintainability. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details. * FEATURE: optimize performance for queries with regexp filters on metric name like `{__name__=~"metric1|...|metricN"}`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1610) from @faceair. * FEATURE: vmui: use Prometheus-compatible query args, so `vmui` could be accessed from graph editor in Grafana. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1619). Thanks to @Loori-R. -* FEATURE: vmselect: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8401,vmstorage2:8401`. This simplifies [manual setup of VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#cluster-setup). -* FEATURE: vminsert: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8400,vmstorage2:8400`. This simplifies [manual setup of VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#cluster-setup). -* FEATURE: add [mad(q)](./MetricsQL.md#mad) function to [MetricsQL](./MetricsQL.md). It calculates [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) for groups of points with identical timestamps across multiple time series. -* FEATURE: add [outliers_mad(tolerance, q)](./MetricsQL.md#outliers_mad) function to [MetricsQL](./MetricsQL.md). It returns time series with peaks outside the [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) multiplied by `tolerance`. -* FEATURE: add `histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)` function to [MetricsQL](./MetricsQL.md). It calculates the given `phi*`-quantiles over the given `buckets` and returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label. -* FEATURE: add `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` function to [MetricsQL](./MetricsQL.md). It calculates the given `phi*`-quantiles over raw samples selected by `series_selector` on the given lookbehind window `d`. It returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label. -* FEATURE: [enterprise](./enterprise.md): do not ask for `-eula` flag if `-version` flag is passed to enterprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621). +* FEATURE: vmselect: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8401,vmstorage2:8401`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup). +* FEATURE: vminsert: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8400,vmstorage2:8400`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup). +* FEATURE: add [mad(q)](https://docs.victoriametrics.com/metricsql/#mad) function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) for groups of points with identical timestamps across multiple time series. +* FEATURE: add [outliers_mad(tolerance, q)](https://docs.victoriametrics.com/metricsql/#outliers_mad) function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It returns time series with peaks outside the [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) multiplied by `tolerance`. +* FEATURE: add `histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates the given `phi*`-quantiles over the given `buckets` and returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label. +* FEATURE: add `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates the given `phi*`-quantiles over raw samples selected by `series_selector` on the given lookbehind window `d`. It returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label. +* FEATURE: [enterprise](https://docs.victoriametrics.com/enterprise/): do not ask for `-eula` flag if `-version` flag is passed to enterprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621). * BUGFIX: properly handle queries with multiple filters matching empty labels such as `metric{label1=~"foo|",label2="bar|"}`. This filter must match the following series: `metric`, `metric{label1="foo"}`, `metric{label2="bar"}` and `metric{label1="foo",label2="bar"}`. Previously it was matching only `metric{label1="foo",label2="bar"}`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601). * BUGFIX: vmselect: reset connection timeouts after each request to `vmstorage`. This should prevent from `cannot read data in 0.000 seconds: unexpected EOF` warning in logs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1562). Thanks to @mxlxm . -* BUGFIX: keep metric name for time series returned from [rollup_candlestick](./MetricsQL.md#rollup_candlestick) function, since the returned series don't change the meaning of the original series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600). -* BUGFIX: use Prometheus-compatible label value formatting for [count_values](./MetricsQL.md#count_values) function. Previously big values could be improperly formatted, which could break query results, which rely on label value such as `... on(label) count_values("label", ...)`. +* BUGFIX: keep metric name for time series returned from [rollup_candlestick](https://docs.victoriametrics.com/metricsql/#rollup_candlestick) function, since the returned series don't change the meaning of the original series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600). +* BUGFIX: use Prometheus-compatible label value formatting for [count_values](https://docs.victoriametrics.com/metricsql/#count_values) function. Previously big values could be improperly formatted, which could break query results, which rely on label value such as `... on(label) count_values("label", ...)`. * BUGFIX: vmagent: properly use `https` scheme for wildcard TLS certificates for `role: ingress` targets in Kubernetes service discovery. See [this issue](https://github.com/prometheus/prometheus/issues/8902). * BUGFIX: vmagent: support host networking mode for `docker_sd_config`. See [this issue](https://github.com/prometheus/prometheus/issues/9116). * BUGFIX: fix non-repeatable results from `quantile_over_time()` function when the number of input samples exceeds 1000. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612). -* BUGFIX: vmagent: fix EC2 zone discovery when `filters` are specified in [ec2_sc_config](./sd_configs.md#ec2_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626). +* BUGFIX: vmagent: fix EC2 zone discovery when `filters` are specified in [ec2_sc_config](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626). ## [v1.65.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.65.0) Released at 2021-09-01 * FEATURE: vmagent: add ability to read scrape configs from multiple files specified in `scrape_config_files` section. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559). -* FEATURE: vmagent: reduce memory usage and CPU usage when [Prometheus staleness tracking](./vmagent.md#prometheus-staleness-markers) is enabled for metrics exported from the deleted or disappeared scrape targets. -* FEATURE: vmagent: add the ability to limit the number of unique time series scraped per each target. This can be done either globally via `-promscrape.seriesLimitPerTarget` command-line option or on per-target basis via `series_limit` option at `scrape_config` section. See [the updated docs on cardinality limiter](./vmagent.md#cardinality-limiter) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561). -* FEATURE: vmagent: discover `role: ingress` and `role: endpointslice` in [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs) via v1 API instead of v1beta1 API if Kubernetes supports it. This fixes service discovery in Kubernetes v1.22 and newer versions. See [these docs](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122). +* FEATURE: vmagent: reduce memory usage and CPU usage when [Prometheus staleness tracking](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) is enabled for metrics exported from the deleted or disappeared scrape targets. +* FEATURE: vmagent: add the ability to limit the number of unique time series scraped per each target. This can be done either globally via `-promscrape.seriesLimitPerTarget` command-line option or on per-target basis via `series_limit` option at `scrape_config` section. See [the updated docs on cardinality limiter](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561). +* FEATURE: vmagent: discover `role: ingress` and `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) via v1 API instead of v1beta1 API if Kubernetes supports it. This fixes service discovery in Kubernetes v1.22 and newer versions. See [these docs](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122). * FEATURE: take into account failed queries in `vm_request_duration_seconds` summary at `/metrics`. Previously only successful queries were taken into account. This could result in skewed summary. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1537). -* FEATURE: vmalert: add an official dashboard for vmalert. See [these docs](./vmalert.md#monitoring). +* FEATURE: vmalert: add an official dashboard for vmalert. See [these docs](https://docs.victoriametrics.com/vmalert/#monitoring). * FEATURE: vmalert: add ability to set additional labels per group via `labels` config section. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471). * FEATURE: vmalert: add `-disableAlertgroupLabel` command-line flag for disabling the label with alert group name. This may be needed for proper deduplication in Alertmanager. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1532). * FEATURE: update Go builder from v1.16.7 to v1.17.0. This improves data ingestion and query performance by up to 5% according to benchmarks. See [the release post for Go1.17](https://go.dev/blog/go1.17). * FEATURE: vmagent: expose `promscrape_discovery_http_errors_total` metric, which can be used for monitoring the number of failed discovery attempts per each `http_sd` config. * FEATURE: do not reset response cache when a sample with old timestamp is ingested into VictoriaMetrics if `-search.disableAutoCacheReset` command-line option is set. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570). -* FEATURE: add `quantiles("phiLabel", phi1, ..., phiN, q)` aggregate function to [MetricsQL](./MetricsQL.md), which calculates the given `phi*` quantiles over time series returned by `q`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573). +* FEATURE: add `quantiles("phiLabel", phi1, ..., phiN, q)` aggregate function to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which calculates the given `phi*` quantiles over time series returned by `q`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573). * BUGFIX: rename `sign` function to `sgn` in order to be consistent with PromQL. See [this pull request from Prometheus](https://github.com/prometheus/prometheus/pull/8457). -* BUGFIX: vmagent: add `role: endpointslice` in [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs) in order to be consistent with Prometheus. Previously this role was supported with incorrect name: `role: endpointslices`. Now both `endpointslice` and `endpointslices` are supported. See [the corresponding code in Prometheus](https://github.com/prometheus/prometheus/blob/2ec6c7dbb82b72834021e01f1773eb90a67a371f/discovery/kubernetes/kubernetes.go#L99). +* BUGFIX: vmagent: add `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) in order to be consistent with Prometheus. Previously this role was supported with incorrect name: `role: endpointslices`. Now both `endpointslice` and `endpointslices` are supported. See [the corresponding code in Prometheus](https://github.com/prometheus/prometheus/blob/2ec6c7dbb82b72834021e01f1773eb90a67a371f/discovery/kubernetes/kubernetes.go#L99). * BUGFIX: improve the detection of the needed free space for background merge operation. This should prevent from possible out of disk space crashes during big merges. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560). * BUGFIX: vmauth: remove trailing slash from the full url before requesting it from the backend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1554). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix timeout error when snapshot takes longer than 10 seconds. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix timeout error when snapshot takes longer than 10 seconds. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571). * BUGFIX: properly parse OpenTSDB `put` messages with multiple spaces between message elements. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1574). Thanks to @envzhu for the fix. ## [v1.64.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.64.1) Released at 2021-08-19 -* FEATURE: add `bitmap_and(q, mask)`, `bitmap_or(q, mask)` and `bitmak_xor(q, mask)` functions to [MetricsQL](./MetricsQL.md). These functions allow performing bitwise operations over data points in time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541). +* FEATURE: add `bitmap_and(q, mask)`, `bitmap_or(q, mask)` and `bitmak_xor(q, mask)` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/). These functions allow performing bitwise operations over data points in time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541). * FEATURE: vmalert: add `-remoteWrite.disablePathAppend` command-line flag, which can be used when custom `-remoteWrite.url` must be specified. For example, `./vmalert -disablePathAppend -remoteWrite.url='http://foo.bar/a/b/c?d=e'` would write data to `http://foo.bar/a/b/c?d=e` instead of `http://foo.bar/a/b/c?d=e/api/v1/write`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1536). * FEATURE: vmagent: add `-promscrape.noStaleMarkers` command-line flag for disabling sending Prometheus stale markers for metrics from disappeared scrape targets. This option may be used for reducing memory usage when scraping big number of metrics with big number of labels and when stale markers aren't needed. * FEATURE: vmselect: add `-search.noStaleMarkers` command-line flag for disabling stale markers handling in queries. This may save some CPU time when the queried data doesn't contain stale markers. @@ -238,7 +238,7 @@ Released at 2021-08-15 * FEATURE: add support for Prometheus staleness markers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526). * FEATURE: vmagent: automatically generate Prometheus staleness markers for the scraped metrics when scrape targets disappear in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526). * FEATURE: add `present_over_time(m[d])` function, which returns 1 if `m` has a least a single sample over the previous duration `d`. This function has been added also to [Prometheus 2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0). -* FEATURE: vmagent: support multitenant writes according to [these docs](./vmagent.md#multitenancy). This allows using a single `vmagent` instance in front of VictoriaMetrics cluster for all the tenants. Thanks to @omarghader for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1505). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491). +* FEATURE: vmagent: support multitenant writes according to [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy). This allows using a single `vmagent` instance in front of VictoriaMetrics cluster for all the tenants. Thanks to @omarghader for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1505). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491). * FEATURE: vmagent: add `__meta_ec2_availability_zone_id` label to discovered Amazon EC2 targets. This label is available in Prometheus [starting from v2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0). * FEATURE: vmagent: add `__meta_gce_interface_ipv4_` labels to discovered GCE targets. These labels are available in Prometheus [starting from v2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0). * FEATURE: add `-search.maxSamplesPerSeries` command-line flag for limiting the number of raw samples a single query can process per each time series. This option can protect from out of memory errors when a query processes tens of millions of raw samples per series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067). @@ -252,7 +252,7 @@ Released at 2021-08-15 * BUGFIX: fix corner cases for queries on time ranges exceeding 40 days. Previously some series can be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486). * BUGFIX: vmselect: return dummy response at `/rules` page in the same way as for `/api/v1/rules` page. The `/rules` page is requested by Grafana 8. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493) for details. * BUGFIX: vmbackup: automatically set default `us-east-1` S3 region if it is missing. This should simplify using S3-compatible services such as MinIO for backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1449). -* BUGFIX: vmselect: prevent from possible deadlock when multiple `target` query args are passed to [Graphite Render API](./README.md#graphite-render-api-usage). +* BUGFIX: vmselect: prevent from possible deadlock when multiple `target` query args are passed to [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). * BUGFIX: return series with `a op b` labels and `N` values for `(a op b) default N` if `(a op b)` returns series with all NaN values. Previously such series were removed. * BUGFIX: vmui: fix layout when the query selects more than 27 time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1497). * BUGFIX: vmagent: restore highlighting in red for DOWN targets at `/targets` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461). @@ -265,8 +265,8 @@ Released at 2021-07-15 * FEATURE: vmselect: embed [vmui](https://github.com/VictoriaMetrics/vmui) into a single-node VictoriaMetrics and into `vmselect` component of cluster version. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413). The web interface is available at the following paths: * `/vmui/` for a single-node VictoriaMetrics * `/select//vmui/` for `vmselect` at cluster version of VictoriaMetrics -* FEATURE: support durations anywhere in [MetricsQL queries](./MetricsQL.md). For example, `sum_over_time(m[1h]) / 1h` is a valid query, which is equivalent to `sum_over_time(m[1h]) / 3600`. -* FEATURE: support durations without suffixes in [MetricsQL queries](./MetricsQL.md). For example, `rate(m[3600])` is a valid query, which is equivalent to `rate(m[1h])`. +* FEATURE: support durations anywhere in [MetricsQL queries](https://docs.victoriametrics.com/metricsql/). For example, `sum_over_time(m[1h]) / 1h` is a valid query, which is equivalent to `sum_over_time(m[1h]) / 3600`. +* FEATURE: support durations without suffixes in [MetricsQL queries](https://docs.victoriametrics.com/metricsql/). For example, `rate(m[3600])` is a valid query, which is equivalent to `rate(m[1h])`. * FEATURE: export `vmselect_request_duration_seconds` and `vminsert_request_duration_seconds` [VictoriaMetrics histograms](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) at `/metrics` page. These histograms can be used for determining latency distribution and SLI/SLO for the served requests. For example, the following query would return the percent of queries that took less than 500ms during the last hour: `histogram_share(500ms, sum(rate(vmselect_request_duration_seconds_bucket[1h])) by (vmrange))`. * FEATURE: vmagent: dynamically reload client TLS certificates from disk on every [mTLS connection](https://developers.cloudflare.com/cloudflare-one/identity/devices/mutual-tls-authentication). This should allow using `vmagent` with [Istio service mesh](https://istio.io/latest/about/service-mesh/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420). * FEATURE: log http request path plus all the query args on errors during request processing. Previously only http request path was logged without query args, so it could be hard debugging such errors. @@ -282,15 +282,15 @@ Released at 2021-07-15 Released at 2021-06-25 -* FEATURE: vmagent: add service discovery for Docker (aka [docker_sd_config](./sd_configs.md#docker_sd_configs)). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1402). -* FEATURE: vmagent: add service discovery for DigitalOcean (aka [digitalocean_sd_config](./sd_configs.md#digitalocean_sd_configs)). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367). +* FEATURE: vmagent: add service discovery for Docker (aka [docker_sd_config](https://docs.victoriametrics.com/sd_configs/#docker_sd_configs)). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1402). +* FEATURE: vmagent: add service discovery for DigitalOcean (aka [digitalocean_sd_config](https://docs.victoriametrics.com/sd_configs/#digitalocean_sd_configs)). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367). * FEATURE: vmagent: change the default value for `-remoteWrite.queues` from 4 to `2 * numCPUs`. This should reduce scrape duration for highly loaded vmagent, which scrapes tens of thousands of targets. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1385). * FEATURE: vmagent: show the number of samples the target returns during the last scrape on `/targets` and `/api/v1/targets` pages. This should simplify debugging targets, which may return too big or too low number of samples. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377). * FEATURE: vmagent: show jobs with zero discovered targets on `/targets` page. This should help debugging improperly configured scrape configs. -* FEATURE: vmagent: support for http-based service discovery (aka [http_sd_config](./sd_configs.md#http_sd_configs)), which has been added since Prometheus 2.28. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1392). +* FEATURE: vmagent: support for http-based service discovery (aka [http_sd_config](https://docs.victoriametrics.com/sd_configs/#http_sd_configs)), which has been added since Prometheus 2.28. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1392). * FEATURE: vmagent: support namespace in Consul service discovery in the same way as Prometheus 2.28 does. See [this issue](https://github.com/prometheus/prometheus/issues/8894) for details. * FEATURE: vmagent: support generic auth configs in `consul_sd_configs` in the same way as Prometheus 2.28 does. See [this issue](https://github.com/prometheus/prometheus/issues/8924) for details. -* FEATURE: [vmctl](./vmctl.md): limit the number of samples per each imported JSON line. This should limit the memory usage at VictoriaMetrics side when importing time series with big number of samples. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): limit the number of samples per each imported JSON line. This should limit the memory usage at VictoriaMetrics side when importing time series with big number of samples. * FEATURE: vmselect: log slow queries across all the `/api/v1/*` handlers (aka [Prometheus query API](https://prometheus.io/docs/prometheus/latest/querying/api)) if their execution duration exceeds `-search.logSlowQueryDuration`. This should simplify debugging slow requests to such handlers as `/api/v1/labels` or `/api/v1/series` additionally to `/api/v1/query` and `/api/v1/query_range`, which were logged in the previous releases. * FEATURE: vminsert: sort the `-storageNode` list in order to guarantee the identical `series -> vmstorage` mapping across all the `vminsert` nodes. This should reduce resource usage (RAM, CPU and disk IO) at `vmstorage` nodes if `vmstorage` addresses are passed in random order to `vminsert` nodes. * FEATURE: vmstorage: reduce memory usage on a system with many CPU cores under high ingestion rate. @@ -298,54 +298,54 @@ Released at 2021-06-25 * BUGFIX: prevent from adding new samples to deleted time series after the rotation of the inverted index (the rotation is performed once per `-retentionPeriod`). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136) for details. * BUGFIX: vmstorage: reduce high disk write IO usage on systems with big number of CPU cores. The issue has been introduced in the release [v1.59.0](#v1590). See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/aa9b56a046b6ae8083fa659df35dd5e994bf9115) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999) for details. * BUGFIX: vmstorage: prevent from incorrect stats collection when multiple concurrent queries execute the same tag filter. This may help reducing CPU usage under certain workloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338). -* BUGFIX: vmselect: return the last timestamp for the max / min value from `tmax_over_time(m[d])` and `tmin_over_time(m[d])` [MetricsQL functions](./MetricsQL.md) as most users expect. See also [this issue](https://github.com/prometheus/prometheus/issues/8966). -* BUGFIX: vmselect: return the expected value for `increase_pure()` [MetricsQL function](./MetricsQL.md) after a gap in a time series. Previously incorrect too big value could be returned after the gap from `increase_pure()`. +* BUGFIX: vmselect: return the last timestamp for the max / min value from `tmax_over_time(m[d])` and `tmin_over_time(m[d])` [MetricsQL functions](https://docs.victoriametrics.com/metricsql/) as most users expect. See also [this issue](https://github.com/prometheus/prometheus/issues/8966). +* BUGFIX: vmselect: return the expected value for `increase_pure()` [MetricsQL function](https://docs.victoriametrics.com/metricsql/) after a gap in a time series. Previously incorrect too big value could be returned after the gap from `increase_pure()`. ## [v1.61.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.1) Released at 2021-06-11 * BUGFIX: vmalert: fix recording rules, which were broken in v1.61.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1369). -* BUGFIX: reset the on-disk cache for mapping from the full metric name to an internal metric id (e.g. `metric_name{labels} -> internal_metric_id`) after deleting metrics via [delete API](./README.md#how-to-delete-time-series). This should prevent from possible inconsistent state after unclean shutdown. This [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347). +* BUGFIX: reset the on-disk cache for mapping from the full metric name to an internal metric id (e.g. `metric_name{labels} -> internal_metric_id`) after deleting metrics via [delete API](https://docs.victoriametrics.com/#how-to-delete-time-series). This should prevent from possible inconsistent state after unclean shutdown. This [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347). ## [v1.61.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.0) Released at 2021-06-09 -* FEATURE: vmalert: add support for backfilling (aka replay) of recording and alerting rules. See [these docs](./vmalert.md#rules-backfilling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836). +* FEATURE: vmalert: add support for backfilling (aka replay) of recording and alerting rules. See [these docs](https://docs.victoriametrics.com/vmalert/#rules-backfilling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836). * FEATURE: vmalert: add a command-line flag `-rule.configCheckInterval` for automatic re-reading of `-rule` files without the need to send SIGHUP signal. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/512). -* FEATURE: vmagent: respect the `sample_limit` and `-promscrape.maxScrapeSize` values when scraping targets in [stream parsing mode](./vmagent.md#stream-parsing-mode). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1331). -* FEATURE: vmauth: add ability to specify mutliple `url_prefix` entries for balancing the load among multiple `vmselect` and/or `vminsert` nodes in a cluster. See [these docs](./vmauth.md#load-balancing). +* FEATURE: vmagent: respect the `sample_limit` and `-promscrape.maxScrapeSize` values when scraping targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1331). +* FEATURE: vmauth: add ability to specify mutliple `url_prefix` entries for balancing the load among multiple `vmselect` and/or `vminsert` nodes in a cluster. See [these docs](https://docs.victoriametrics.com/vmauth/#load-balancing). * FEATURE: vminsert: add `-disableRerouting` command-line flag for forcibly disabling the rerouting. This should help resolving [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054) issues. * FEATURE: vminsert: reduce the probability of global re-routing storm if all the vmstorage nodes cannot keep up with the given ingestion rate for some time. This should improve cluster stability in such cases. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054) issues. * FEATURE: allow building VictoriaMetrics components for Solaris / SmartOS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322). -* FEATURE: vmagent: add ability to debug relabeling rules. See [these docs](./vmagent.md#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343). +* FEATURE: vmagent: add ability to debug relabeling rules. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343). * BUGFIX: reduce CPU usage by up to 2x during querying a database with big number of active daily time series. The issue has been introduced in `v1.59.0`. * BUGFIX: vmagent: properly apply auth and tls configs in `eureka_sd_configs`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1350). * BUGFIX: vmauth: do not panic on aborted http requests. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353). -* BUGFIX: properly generate `target` property for `*Series(foo.*.bar)` responses returned from [Graphite Render API](./README.md#graphite-render-api-usage). Previously the `target` contained the expanded list of series for `foo.*.bar`, e.g. `sumSeries(foo.a.bar,foo.b.bar,...foo.z.bar)`. Now VictoriaMetrics returns `sumSeries(foo.*.bar)` as a target in the same way as Graphite does. +* BUGFIX: properly generate `target` property for `*Series(foo.*.bar)` responses returned from [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously the `target` contained the expanded list of series for `foo.*.bar`, e.g. `sumSeries(foo.a.bar,foo.b.bar,...foo.z.bar)`. Now VictoriaMetrics returns `sumSeries(foo.*.bar)` as a target in the same way as Graphite does. ## [v1.60.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.60.0) Released at 2021-05-24 -* FEATURE: add ability to limit the number of unique time series, which can be added to storage per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](./README.md#cardinality-limiter). -* FEATURE: vmagent: add ability to limit the number of unique time series, which can be sent to remote storage systems per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](./vmagent.md#cardinality-limiter). -* FEATURE: vmalert: add ability to run alerting and recording rules for multiple tenants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740) and [these docs](./vmalert.md#multitenancy). -* FEATURE: vminsert: add support for data ingestion via other `vminsert` nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of `vminsert` nodes to another level of `vminsert` nodes. See [these docs](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/541#issuecomment-835487858) for details. +* FEATURE: add ability to limit the number of unique time series, which can be added to storage per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/#cardinality-limiter). +* FEATURE: vmagent: add ability to limit the number of unique time series, which can be sent to remote storage systems per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). +* FEATURE: vmalert: add ability to run alerting and recording rules for multiple tenants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740) and [these docs](https://docs.victoriametrics.com/vmalert/#multitenancy). +* FEATURE: vminsert: add support for data ingestion via other `vminsert` nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of `vminsert` nodes to another level of `vminsert` nodes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/541#issuecomment-835487858) for details. * FEATURE: vmagent: reload `bearer_token_file`, `credentials_file` and `password_file` contents every second. This allows dynamically changing the contents of these files during target scraping and service discovery without the need to restart `vmagent`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297). * FEATURE: vmalert: add a flag to control behaviour on startup for state restore errors. Such errors were returned and logged before as well. Now user can specify whether to just log these errors (`-remoteRead.ignoreRestoreErrors=true`) or to stop the process (`-remoteRead.ignoreRestoreErrors=false`). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252). * FEATURE: vmalert: add ability to pass `round_digits` query arg to datasource via `-datasource.roundDigits` command-line flag. This can be used for limiting the number of decimal digits after the point in recording rule results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525). * FEATURE: return `X-Server-Hostname` header in http responses of all the VictoriaMetrics components. This should simplify tracing the origin server behind a load balancer or behind auth proxy during troubleshooting. -* FEATURE: vmselect: allow to use 2x more memory for query processing at `vmselect` nodes in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md). This should allow processing heavy queries without the need to increase RAM size at `vmselect` nodes. -* FEATURE: add ability to filter `/api/v1/status/tsdb` output with arbitrary [time series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) passed via `match[]` query args. See [these docs](./README.md#tsdb-stats) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168) for details. +* FEATURE: vmselect: allow to use 2x more memory for query processing at `vmselect` nodes in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). This should allow processing heavy queries without the need to increase RAM size at `vmselect` nodes. +* FEATURE: add ability to filter `/api/v1/status/tsdb` output with arbitrary [time series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) passed via `match[]` query args. See [these docs](https://docs.victoriametrics.com/#tsdb-stats) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168) for details. * FEATURE: automatically detect memory and cpu limits for VictoriaMetrics components running under [cgroup v2](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) environments such as [HashiCorp Nomad](https://www.nomadproject.io/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269). * FEATURE: vmauth: allow `-auth.config` reloading via `/-/reload` http endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194). -* FEATURE: add `timezone_offset(tz)` function. It returns offset in seconds for the given timezone `tz` relative to UTC. This can be useful when combining with datetime-related functions. For example, `day_of_week(time()+timezone_offset("America/Los_Angeles"))` would return weekdays for `America/Los_Angeles` time zone. Special `Local` time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306) and [MetricsQL docs](./MetricsQL.md) for more details. +* FEATURE: add `timezone_offset(tz)` function. It returns offset in seconds for the given timezone `tz` relative to UTC. This can be useful when combining with datetime-related functions. For example, `day_of_week(time()+timezone_offset("America/Los_Angeles"))` would return weekdays for `America/Los_Angeles` time zone. Special `Local` time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306) and [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for more details. * FEATURE: vmagent: add support for OAuth2 authorization for scrape targets and service discovery in the same way as Prometheus does. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2). * FEATURE: vmagent: add support for OAuth2 authorization when writing data to `-remoteWrite.url`. See `-remoteWrite.oauth2.*` config params in `/path/to/vmagent -help` output. -* FEATURE: vmalert: add ability to set `extra_filter_labels` at alerting and recording group configs. See [these docs](./vmalert.md#groups). +* FEATURE: vmalert: add ability to set `extra_filter_labels` at alerting and recording group configs. See [these docs](https://docs.victoriametrics.com/vmalert/#groups). * FEATURE: vmstorage: reduce memory usage by up to 30% when ingesting big number of active time series. * BUGFIX: vmagent: do not retry scraping targets, which don't support HTTP. This should reduce CPU load and network usage at `vmagent` and at scrape target. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289). @@ -368,8 +368,8 @@ Released at 2021-05-01 * FEATURE: improved new time series registration speed on systems with many CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244). Thanks to @waldoweng for the idea and [draft implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243). * FEATURE: vmalert: use the same technique as Grafana for determining evaluation timestamps for recording rules. This should make consistent graphs for series generated by recording rules compared to graphs generated for queries from recording rules in Grafana. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232). -* FEATURE: vmauth: add ability to set mandatory query args in `url_prefix`. For example, `url_prefix: http://vm:8428/?extra_label=team=dev` would add `extra_label=team=dev` query arg to all the incoming requests. See [the example](./vmauth.md#auth-config) for more details. -* FEATURE: vmctl: add OpenTSDB migration option. See more details [here](./vmctl.md#migrating-data-from-opentsdb). +* FEATURE: vmauth: add ability to set mandatory query args in `url_prefix`. For example, `url_prefix: http://vm:8428/?extra_label=team=dev` would add `extra_label=team=dev` query arg to all the incoming requests. See [the example](https://docs.victoriametrics.com/vmauth/#auth-config) for more details. +* FEATURE: vmctl: add OpenTSDB migration option. See more details [here](https://docs.victoriametrics.com/vmctl#migrating-data-from-opentsdb). Thanks to @johnseekins! * FEATURE: log metrics with dropped labels if the number of labels in the ingested metric exceeds `-maxLabelsPerTimeseries`. This should simplify debugging for this case. * FEATURE: vmagent: list user-visible endpoints at `http://vmagent:8429/`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1251). @@ -384,8 +384,8 @@ Thanks to @johnseekins! Released at 2021-04-08 * FEATURE: vminsert and vmagent: add `-sortLabels` command-line flag for sorting metric labels before pushing them to `vmstorage`. This should reduce the size of `MetricName -> internal_series_id` cache (aka `vm_cache_size_bytes{type="storage/tsid"}`) when ingesting samples for the same time series with distinct order of labels. For example, `foo{k1="v1",k2="v2"}` and `foo{k2="v2",k1="v1"}` represent a single time series. Labels sorting is disabled by default, since the majority of established exporters preserve the order of labels for the exported metrics. -* FEATURE: allow specifying label value alongside label name for the `others sum` time series returned from `topk_*` and `bottomk_*` functions from [MetricsQL](./MetricsQL.md). For example, `topk_avg(3, max(process_resident_memory_bytes) by (instance), "instance=other_sum")` would return top 3 series from `max(process_resident_memory_bytes) by (instance)` plus a series containing the sum of other series. The `others sum` series will have `{instance="other_sum"}` label. -* FEATURE: do not delete `dst_label` when applying `label_copy(q, "src_label", "dst_label")` and `label_move(q, "src_label", "dst_label")` to series without `src_label` and with non-empty `dst_label`. See more details at [MetricsQL docs](./MetricsQL.md). +* FEATURE: allow specifying label value alongside label name for the `others sum` time series returned from `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/metricsql/). For example, `topk_avg(3, max(process_resident_memory_bytes) by (instance), "instance=other_sum")` would return top 3 series from `max(process_resident_memory_bytes) by (instance)` plus a series containing the sum of other series. The `others sum` series will have `{instance="other_sum"}` label. +* FEATURE: do not delete `dst_label` when applying `label_copy(q, "src_label", "dst_label")` and `label_move(q, "src_label", "dst_label")` to series without `src_label` and with non-empty `dst_label`. See more details at [MetricsQL docs](https://docs.victoriametrics.com/metricsql/). * FEATURE: update Go builder from `v1.16.2` to `v1.16.3`. This should fix [these issues](https://github.com/golang/go/issues?q=milestone%3AGo1.16.3+label%3ACherryPickApproved). * FEATURE: vmagent: add support for `follow_redirects` option to `scrape_configs` section in the same way as [Prometheus 2.26 does](https://github.com/prometheus/prometheus/pull/8546). * FEATURE: vmagent: add support for `authorization` section in `-promscrape.config` in the same way as [Prometheus 2.26 does](https://github.com/prometheus/prometheus/pull/8512). @@ -396,11 +396,11 @@ Released at 2021-04-08 * FEATURE: vmagent: add support for `proxy_tls_config`, `proxy_authorization`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options in `consul_sd_config`, `dockerswarm_sd_config` and `eureka_sd_config` sections. * FEATURE: vmagent: pass `X-Prometheus-Scrape-Timeout-Seconds` header to scrape targets as Prometheus does. In this case scrape targets can limit the time needed for performing the scrape. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179#issuecomment-813118733) for details. * FEATURE: vmagent: drop corrupted persistent queue files at `-remoteWrite.tmpDataPath` instead of throwing a fatal error. Corrupted files can appear after unclean shutdown of `vmagent` such as OOM kill or hardware reset. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030). -* FEATURE: vmauth: add support for authorization via [bearer token](https://swagger.io/docs/specification/authentication/bearer-authentication/). See [the docs](./vmauth.md#auth-config) for details. +* FEATURE: vmauth: add support for authorization via [bearer token](https://swagger.io/docs/specification/authentication/bearer-authentication/). See [the docs](https://docs.victoriametrics.com/vmauth/#auth-config) for details. * FEATURE: publish `arm64` and `amd64` binaries for cluster version of VictoriaMetrics at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). * BUGFIX: properly handle `/api/v1/labels` and `/api/v1/label//values` queries on big `start ... end` time range. This should fix big resource usage when VictoriaMetrics is queried with [Promxy](https://github.com/jacksontj/promxy) v0.0.62 or newer versions. -* BUGFIX: do not break sort order for series returned from `topk*`, `bottomk*` and `outliersk` [MetricsQL](./MetricsQL.md) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189). +* BUGFIX: do not break sort order for series returned from `topk*`, `bottomk*` and `outliersk` [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189). * BUGFIX: vmagent: properly work with simple HTTP proxies which don't support `CONNECT` method. For example, [PushProx](https://github.com/prometheus-community/PushProx). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179). * BUGFIX: vmagent: properly discover targets if multiple namespace selectors are put inside `kubernetes_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170). * BUGFIX: vmagent: properly discover `role: endpoints` and `role: endpointslices` targets in `kubernetes_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182). @@ -427,37 +427,37 @@ Released at 2021-03-29 * `process_resident_memory_shared_bytes` - RSS share for memory shared with other processes (aka shared memory). This share can be freed by the OS at any time, so it must be ignored by OOM killer. * `process_resident_memory_peak_bytes` - peak RSS usage for the process. * `process_virtual_memory_peak_bytes` - peak virtual memory usage for the process. -* FEATURE: accept and enforce `extra_label==` query arg at [Graphite APIs](./README.md#graphite-api-usage). +* FEATURE: accept and enforce `extra_label==` query arg at [Graphite APIs](https://docs.victoriametrics.com/#graphite-api-usage). * FEATURE: use InfluxDB field as metric name if measurement is empty and `-influxSkipSingleField` command-line is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139). * FEATURE: vmagent: add `-promscrape.consul.waitTime` command-line flag for tuning the maximum wait time for Consul service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144). * FEATURE: vmagent: add `vm_promscrape_discovery_kubernetes_stale_resource_versions_total` metric for monitoring the frequency of `too old resource version` errors during Kubernetes service discovery. -* FEATURE: single-node VictoriaMetrics: log metrics with timestamps older than `-search.cacheTimestampOffset` compared to the current time. See [these docs](./README.md#backfilling) for details. +* FEATURE: single-node VictoriaMetrics: log metrics with timestamps older than `-search.cacheTimestampOffset` compared to the current time. See [these docs](https://docs.victoriametrics.com/#backfilling) for details. * BUGFIX: prevent from infinite loop on `{__graphite__="..."}` filters when a metric name contains `*`, `{` or `[` chars. -* BUGFIX: prevent from infinite loop in `/metrics/find` and `/metrics/expand` [Graphite Metrics API handlers](./README.md#graphite-metrics-api-usage) when they match metric names or labels with `*`, `{` or `[` chars. +* BUGFIX: prevent from infinite loop in `/metrics/find` and `/metrics/expand` [Graphite Metrics API handlers](https://docs.victoriametrics.com/#graphite-metrics-api-usage) when they match metric names or labels with `*`, `{` or `[` chars. * BUGFIX: do not merge duplicate time series during requests to `/api/v1/query`. See * BUGFIX: vmagent: properly handle `too old resource version` error messages from Kubernetes watch API. See * BUGFIX: vmagent: do not retry sending data blocks if remote storage returns `400 Bad Request` error. The number of dropped blocks due to such errors can be monitored with `vmagent_remotewrite_packets_dropped_total` metrics. See -* BUGFIX: properly calculate `summarize` and `*Series` functions in [Graphite Render API](./README.md#graphite-render-api-usage). +* BUGFIX: properly calculate `summarize` and `*Series` functions in [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). ## [v1.56.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.56.0) Released at 2021-03-17 -* FEATURE: add the following functions to [MetricsQL](./MetricsQL.md): +* FEATURE: add the following functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/): * `histogram_avg(buckets)` - returns the average value for the given buckets. * `histogram_stdvar(buckets)` - returns standard variance for the given buckets. * `histogram_stddev(buckets)` - returns standard deviation for the given buckets. * FEATURE: export `vm_available_memory_bytes` and `vm_available_cpu_cores` metrics, which show the number of available RAM and available CPU cores for VictoriaMetrics apps. * FEATURE: export `vm_index_search_duration_seconds` histogram, which can be used for troubleshooting time series search performance. -* FEATURE: vmagent: add ability to replicate scrape targets among `vmagent` instances in the cluster with `-promscrape.cluster.replicationFactor` command-line flag. See [these docs](./vmagent.md#scraping-big-number-of-targets). -* FEATURE: vmagent: accept `scrape_offset` option at `scrape_config`. This option may be useful when scrapes must start at the specified offset of every scrape interval. See [these docs](./vmagent.md#troubleshooting) for details. -* FEATURE: vmagent: support `proxy_tls_config`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options at `scrape_config` section for configuring proxies specified via `proxy_url`. See [these docs](./vmagent.md#scraping-targets-via-a-proxy). +* FEATURE: vmagent: add ability to replicate scrape targets among `vmagent` instances in the cluster with `-promscrape.cluster.replicationFactor` command-line flag. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). +* FEATURE: vmagent: accept `scrape_offset` option at `scrape_config`. This option may be useful when scrapes must start at the specified offset of every scrape interval. See [these docs](https://docs.victoriametrics.com/vmagent/#troubleshooting) for details. +* FEATURE: vmagent: support `proxy_tls_config`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options at `scrape_config` section for configuring proxies specified via `proxy_url`. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-targets-via-a-proxy). * FEATURE: vmauth: allow using regexp paths in `url_map`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112) for details. * FEATURE: accept `round_digits` query arg at `/api/v1/query` and `/api/v1/query_range` handlers. This option can be set at Prometheus datasource in Grafana for limiting the number of digits after the decimal point in response values. * FEATURE: add `-influx.databaseNames` command-line flag, which can be used for accepting data from some Telegraf plugins such as [fluentd plugin](https://github.com/fangli/fluent-plugin-influxdb). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124). * FEATURE: add `-logNewSeries` command-line flag, which can be used for debugging the source of time series churn rate. -* FEATURE: publish Windows builds for [vmagent](./vmagent.md), [vmalert](./vmalert.md), [vmauth](./vmauth.md) and [vmctl](./vmctl.md) at `vmutils-windows-*.zip` archives at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). +* FEATURE: publish Windows builds for [vmagent](https://docs.victoriametrics.com/vmagent/), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmctl](https://docs.victoriametrics.com/vmctl/) at `vmutils-windows-*.zip` archives at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). * FEATURE: listen for IPv6 UDP if `-enableTCP6` command-line flag is passed to VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131). * BUGFIX: vmagent: prevent from high CPU usage bug during failing scrapes with small `scrape_timeout` (less than a few seconds). @@ -481,8 +481,8 @@ Released at 2021-03-03 Released at 2021-03-02 -* FEATURE: add `sign(q)` and `clamp(q, min, max)` functions, which are planned to be added in [the upcoming Prometheus release](https://twitter.com/roidelapluie/status/1363428376162295811) . The `last_over_time(m[d])` function is already supported in [MetricsQL](./MetricsQL.md). -* FEATURE: vmagent: add `scrape_align_interval` config option, which can be used for aligning scrapes to the beginning of the configured interval. See [these docs](./vmagent.md#troubleshooting) for details. +* FEATURE: add `sign(q)` and `clamp(q, min, max)` functions, which are planned to be added in [the upcoming Prometheus release](https://twitter.com/roidelapluie/status/1363428376162295811) . The `last_over_time(m[d])` function is already supported in [MetricsQL](https://docs.victoriametrics.com/metricsql/). +* FEATURE: vmagent: add `scrape_align_interval` config option, which can be used for aligning scrapes to the beginning of the configured interval. See [these docs](https://docs.victoriametrics.com/vmagent/#troubleshooting) for details. * FEATURE: expose io-related metrics at `/metrics` page for every VictoriaMetrics component: * `process_io_read_bytes_total` - the number of bytes read via io syscalls such as read and pread * `process_io_written_bytes_total` - the number of bytes written via io syscalls such as write and pwrite @@ -490,12 +490,12 @@ Released at 2021-03-02 * `process_io_write_syscalls_total` - the number of write syscalls such as write and pwrite * `process_io_storage_read_bytes_total` - the number of bytes read from storage layer * `process_io_storage_written_bytes_total` - the number of bytes written to storage layer -* FEATURE: vmagent: add ability to spread scrape targets among multiple `vmagent` instances. See [these docs](./vmagent.md#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084) for details. +* FEATURE: vmagent: add ability to spread scrape targets among multiple `vmagent` instances. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084) for details. * FEATURE: vmagent: use watch API for Kubernetes service discovery. This should reduce load on Kubernetes API server when it tracks big number of objects (for example, 10K pods). This should also reduce the time needed for k8s targets discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1057) for details. * FEATURE: vmagent: export `vm_promscrape_target_relabel_duration_seconds` metric, which can be used for monitoring the time spend on relabeling for discovered targets. -* FEATURE: vmagent: optimize [relabeling](./vmagent.md#relabeling) performance for common cases. +* FEATURE: vmagent: optimize [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) performance for common cases. * FEATURE: add `increase_pure(m[d])` function to MetricsQL. It works the same as `increase(m[d])` except of various edge cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) for details. -* FEATURE: increase accuracy for `buckets_limit(limit, buckets)` results for small `limit` values. See [MetricsQL docs](./MetricsQL.md) for details. +* FEATURE: increase accuracy for `buckets_limit(limit, buckets)` results for small `limit` values. See [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for details. * FEATURE: vmagent: initial support for Windows build with `CGO_ENABLED=0 GOOS=windows go build ./app/vmagent`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036). * FEATURE: vmagent: support WebIdentityToken auth in EC2 service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1080) for details. * FEATURE: vmalert: properly process query params in `-datasource.url` and `-remoteRead.url` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1087) for details. @@ -526,14 +526,14 @@ Released at 2021-02-18 * FEATURE: optimize searching for matching metrics for `metric{}` queries if `` contains at least a single filter. For example, the query `up{job="foobar"}` should find the matching time series much faster than previously. * FEATURE: reduce execution times for `q1 q2` queries by executing `q1` and `q2` in parallel. * FEATURE: switch from Go1.15 to [Go1.16](https://golang.org/doc/go1.16) for building prod binaries. -* FEATURE: single-node VictoriaMetrics now accepts requests to handlers with `/prometheus` and `/graphite` prefixes such as `/prometheus/api/v1/query`. This improves compatibility with [handlers from VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format). +* FEATURE: single-node VictoriaMetrics now accepts requests to handlers with `/prometheus` and `/graphite` prefixes such as `/prometheus/api/v1/query`. This improves compatibility with [handlers from VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format). * FEATURE: expose `process_open_fds` and `process_max_fds` metrics. These metrics can be used for alerting when `process_open_fds` reaches `process_max_fds`. See and -* FEATURE: vmalert: add `-datasource.appendTypePrefix` command-line option for querying both Prometheus and Graphite datasource in cluster version of VictoriaMetrics. See [these docs](./vmalert.md#graphite) for details. +* FEATURE: vmalert: add `-datasource.appendTypePrefix` command-line option for querying both Prometheus and Graphite datasource in cluster version of VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/vmalert/#graphite) for details. * FEATURE: vmauth: add ability to route requests from a single user to multiple destinations depending on the requested paths. See * FEATURE: remove dependency on external programs such as `cat`, `grep` and `cut` when detecting cpu and memory limits inside Docker or LXC container. * FEATURE: vmagent: add `__meta_kubernetes_endpoints_label_*`, `__meta_kubernetes_endpoints_labelpresent_*`, `__meta_kubernetes_endpoints_annotation_*` and `__meta_kubernetes_endpoints_annotationpresent_*` labels for `role: endpoints` in Kubernetes service discovery. These labels where added in Prometheus 2.25. * FEATURE: reduce the minimum supported retention period for inverted index (aka `indexdb`) from one month to one day. This should reduce disk space usage for `<-storageDataPath>/indexdb` folder if `-retentionPeriod` is set to values smaller than one month. -* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](./enterprise.md). See for details. +* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). See for details. * BUGFIX: properly convert regexp tag filters containing escaped dots to non-regexp tag filters. For example, `{foo=~"bar\.baz"}` should be converted to `{foo="bar.baz"}`. Previously it was incorrectly converted to `{foo="bar\.baz"}`, which could result in missing time series for this tag filter. * BUGFIX: do not spam error logs when discovering Docker Swarm targets without dedicated IP. See . @@ -551,14 +551,14 @@ Released at 2021-02-03 Released at 2021-02-03 -* FEATURE: added [vmctl tool](./vmctl.md) to VictoriaMetrics release process. Now it is packaged in `vmutils-*.tar.gz` archive on [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Source code for `vmctl` tool has been moved from [github.com/VictoriaMetrics/vmctl](https://github.com/VictoriaMetrics/vmctl) to [github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmctl). +* FEATURE: added [vmctl tool](https://docs.victoriametrics.com/vmctl/) to VictoriaMetrics release process. Now it is packaged in `vmutils-*.tar.gz` archive on [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Source code for `vmctl` tool has been moved from [github.com/VictoriaMetrics/vmctl](https://github.com/VictoriaMetrics/vmctl) to [github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmctl). * FEATURE: added `-loggerTimezone` command-line flag for adjusting time zone for timestamps in log messages. By default UTC is used. * FEATURE: added `-search.maxStepForPointsAdjustment` command-line flag, which can be used for disabling adjustment for points returned by `/api/v1/query_range` handler if such points have timestamps closer than `-search.latencyOffset` to the current time. Such points may contain incomplete data, so they are substituted by the previous values for `step` query args smaller than one minute by default. * FEATURE: vmselect: added ability to use Graphite-compatible filters in MetricsQL via `{__graphite__="foo.*.bar"}` syntax. This expression is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. This feature deprecates the usage of `-search.treatDotsAsIsInRegexps` command-line flag. -* FEATURE: vmselect: added ability to set additional label filters, which must be applied during queries. Such label filters can be set via optional `extra_label` query arg, which is accepted by [querying API](./README.md#prometheus-querying-api-usage) handlers. For example, the request to `/api/v1/query_range?extra_label=tenant_id=123&query=` adds `{tenant_id="123"}` label filter to the given ``. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting +* FEATURE: vmselect: added ability to set additional label filters, which must be applied during queries. Such label filters can be set via optional `extra_label` query arg, which is accepted by [querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) handlers. For example, the request to `/api/v1/query_range?extra_label=tenant_id=123&query=` adds `{tenant_id="123"}` label filter to the given ``. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. [Contact us](mailto:sales@victoriametrics.com) if you need assistance with such a proxy. See . * FEATURE: vmalert: added `-datasource.queryStep` command-line flag for passing optional `step` query arg to `/api/v1/query` endpoint. See -* FEATURE: vmalert: added ability to query Graphite datasource when evaluating alerting and recording rules. See [these docs](./vmalert.md#graphite) for details. +* FEATURE: vmalert: added ability to query Graphite datasource when evaluating alerting and recording rules. See [these docs](https://docs.victoriametrics.com/vmalert/#graphite) for details. * FEATURE: vmagent: added `-remoteWrite.roundDigits` command-line option for rounding metric values to the given number of decimal digits after the point before sending the metric to the corresponding `-remoteWrite.url`. This option can be used for improving data compression on the remote storage, because values with lower number of decimal digits can be compressed better than values with bigger number of decimal digits. * FEATURE: vmagent: added `-remoteWrite.rateLimit` command-line flag for limiting data transfer rate to `-remoteWrite.url`. This may be useful when big amounts of buffered data is sent after temporarily unavailability of the remote storage. See * FEATURE: vmagent: export the following additional metrics, which may be useful during troubleshooting: @@ -568,7 +568,7 @@ in front of VictoriaMetrics. [Contact us](mailto:sales@victoriametrics.com) if y * `vm_promscrape_discovery_retries_total` * `vm_promscrape_scrape_retries_total` * `vm_promscrape_service_discovery_duration_seconds` -* FEATURE: vmselect: initial implementation for [Graphite Render API](./README.md#graphite-render-api-usage). +* FEATURE: vmselect: initial implementation for [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). * BUGFIX: vmagent: reduce HTTP reconnection rate for scrape targets. Previously vmagent could erroneously close HTTP keep-alive connections more frequently than needed. * BUGFIX: vmagent: retry scrape and service discovery requests when the remote server closes HTTP keep-alive connection. Previously `disable_keepalive: true` option could be used under `scrape_configs` section when working with such servers. @@ -579,10 +579,10 @@ Released at 2021-01-13 * FEATURE: provide a sample list of alerting rules for VictoriaMetrics components. It is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts). * FEATURE: disable final merge for data for the previous month at the beginning of new month, since it may result in high disk IO and CPU usage. Final merge can be enabled by setting `-finalMergeDelay` command-line flag to positive duration. -* FEATURE: add `tfirst_over_time(m[d])` and `tlast_over_time(m[d])` functions to [MetricsQL](./MetricsQL.md) for returning timestamps for the first and the last data point in `m` over `d` duration. +* FEATURE: add `tfirst_over_time(m[d])` and `tlast_over_time(m[d])` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/) for returning timestamps for the first and the last data point in `m` over `d` duration. * FEATURE: add ability to pass multiple labels to `sort_by_label()` and `sort_by_label_desc()` functions. See . * FEATURE: enforce at least TLS v1.2 when accepting HTTPS requests if `-tls`, `-tlsCertFile` and `-tlsKeyFile` command-line flags are set, because older TLS protocols such as v1.0 and v1.1 have been deprecated due to security vulnerabilities. -* FEATURE: support `extra_label` query arg for all HTTP-based [data ingestion protocols](./README.md#how-to-import-time-series-data). This query arg can be used for specifying extra labels which should be added for the ingested data. +* FEATURE: support `extra_label` query arg for all HTTP-based [data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data). This query arg can be used for specifying extra labels which should be added for the ingested data. * FEATURE: vmbackup: increase backup chunk size from 128MB to 1GB. This should reduce the number of Object storage API calls during backups by 8x. This may also reduce costs, since object storage API calls usually have non-zero costs. See and . * BUGFIX: properly parse escaped unicode chars in MetricsQL metric names, label names and function names. See @@ -596,4 +596,4 @@ Released at 2021-01-13 ## Previous releases -See changes for older releases [here](./CHANGELOG_2020.md). +See changes for older releases [here](https://docs.victoriametrics.com/changelog_2020/). diff --git a/docs/CHANGELOG_2022.md b/docs/CHANGELOG_2022.md index 2fa4f18b5..f3ad59fb6 100644 --- a/docs/CHANGELOG_2022.md +++ b/docs/CHANGELOG_2022.md @@ -13,23 +13,23 @@ aliases: Released at 2022-12-20 -**Update note 1:** This and newer releases of VictoriaMetrics may return gaps for `rate(m[d])` queries on short time ranges if `[d]` lookbehind window is set explicitly. For example, `rate(http_requests_total[$__interval])`. This reduces confusion level when the user expects the needed results from the query with explicitly set lookbehind window. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483). The previous gap filling behaviour can be restored by removing explicit lookbehind window `[d]` from the query, e.g. by substituting the `rate(m[d])` with `rate(m)`. See [these docs](./MetricsQL.md#implicit-query-conversions) for details. +**Update note 1:** This and newer releases of VictoriaMetrics may return gaps for `rate(m[d])` queries on short time ranges if `[d]` lookbehind window is set explicitly. For example, `rate(http_requests_total[$__interval])`. This reduces confusion level when the user expects the needed results from the query with explicitly set lookbehind window. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483). The previous gap filling behaviour can be restored by removing explicit lookbehind window `[d]` from the query, e.g. by substituting the `rate(m[d])` with `rate(m)`. See [these docs](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions) for details. -* BUGFIX: fix `error when searching for TSIDs by metricIDs in the previous indexdb: EOF` error, which can occur during queries after unclean shutdown of VictoriaMetrics (e.g. via hardware reset, out of memory crash or `kill -9`). The error has been introduced in [v1.85.2](./CHANGELOG.md#v1852). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515). -* BUGFIX: [VictoriaMetrics enterprise](./enterprise.md): expose proper values for `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which were added at [v1.78.0](./CHANGELOG.md#v1780). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). -* BUGFIX: [MetricsQL](./MetricsQL.md): never extend explicitly set lookbehind window for [rate()](./MetricsQL.md#rate) function. This reduces the level of confusion when the user expects the needed results after explicitly seting the lookbehind window `[d]` in the query `rate(m[d])`. Previously VictoriaMetrics could silently extend the lookbehind window, so it covers at least two raw samples. Now this behavior works only if the lookbehind window in square brackets isn't set explicitly, e.g. in the case of `rate(m)`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483) for details. -* BUGFIX: [vmagent](./vmagent.md): respect `-usePromCompatibleNaming` flag if no relabeling or extra labels were set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3511) for details. -* BUGFIX: [vmui](./README.md#vmui): fix the wrong legend when queries are hidden. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3512). -* BUGFIX: [vmui](./README.md#vmui): fix incorrect time selection after the timezone change. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3519). +* BUGFIX: fix `error when searching for TSIDs by metricIDs in the previous indexdb: EOF` error, which can occur during queries after unclean shutdown of VictoriaMetrics (e.g. via hardware reset, out of memory crash or `kill -9`). The error has been introduced in [v1.85.2](https://docs.victoriametrics.com/changelog/#v1852). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515). +* BUGFIX: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): expose proper values for `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which were added at [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): never extend explicitly set lookbehind window for [rate()](https://docs.victoriametrics.com/metricsql/#rate) function. This reduces the level of confusion when the user expects the needed results after explicitly seting the lookbehind window `[d]` in the query `rate(m[d])`. Previously VictoriaMetrics could silently extend the lookbehind window, so it covers at least two raw samples. Now this behavior works only if the lookbehind window in square brackets isn't set explicitly, e.g. in the case of `rate(m)`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): respect `-usePromCompatibleNaming` flag if no relabeling or extra labels were set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3511) for details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the wrong legend when queries are hidden. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3512). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix incorrect time selection after the timezone change. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3519). ## [v1.85.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.2) Released at 2022-12-19 -* FEATURE: support overriding of `-search.latencyOffset` value via URL param `latency_offset` when performing requests to [/api/v1/query](./keyConcepts.md#instant-query) and [/api/v1/query_range](./keyConcepts.md#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481). +* FEATURE: support overriding of `-search.latencyOffset` value via URL param `latency_offset` when performing requests to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481). * FEATURE: allow changing field names in JSON logs if VictoriaMetrics components are started with `-loggerFormat=json` command-line flags. The field names can be changed with the `-loggerJSONFields` command-line flag. For example `-loggerJSONFields=ts:timestamp,msg:message` would rename `ts` and `msg` fields on the output JSON to `timestamp` and `message` fields. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348). Thanks to @michal-kralik for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3488). -* FEATURE: [vmagent](./vmagent.md): expose `__meta_consul_tag_` and `__meta_consul_tagpresent_` labels for targets discovered via [consul_sd_configs](./sd_configs.md#consul_sd_configs). This simplifies converting [Consul service tags](https://developer.hashicorp.com/consul/docs/services/discovery/dns-overview) to target labels with a simple [relabeling rule](./vmagent.md#relabeling): +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `__meta_consul_tag_` and `__meta_consul_tagpresent_` labels for targets discovered via [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs). This simplifies converting [Consul service tags](https://developer.hashicorp.com/consul/docs/services/discovery/dns-overview) to target labels with a simple [relabeling rule](https://docs.victoriametrics.com/vmagent/#relabeling): ```yaml - action: labelmap @@ -38,41 +38,41 @@ Released at 2022-12-19 This resolves [this StackOverflow question](https://stackoverflow.com/questions/44339461/relabeling-in-prometheus). -* BUGFIX: properly return query results for time series, which stop receiving new samples after the rotation of `indexdb`. Previously such time series could be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502). The issue has been introduced in [v1.83.0](./CHANGELOG.md#v1830). +* BUGFIX: properly return query results for time series, which stop receiving new samples after the rotation of `indexdb`. Previously such time series could be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502). The issue has been introduced in [v1.83.0](https://docs.victoriametrics.com/changelog/#v1830). * BUGFIX: allow specifying values bigger than 2GiB to the following command-line flag values on 32-bit architectures (`386` and `arm`): `-storage.minFreeDiskSpaceBytes` and `-remoteWrite.maxDiskUsagePerURL`. Previously values bigger than 2GiB were incorrectly truncated on these architectures. -* BUGFIX: [vmagent](./vmagent.md): stop dropping metric name by a mistake on the [/metric-relabel-debug](./vmagent.md#relabel-debug) page. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): stop dropping metric name by a mistake on the [/metric-relabel-debug](https://docs.victoriametrics.com/vmagent/#relabel-debug) page. ## [v1.85.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.1) Released at 2022-12-14 -**It is recommended upgrading to [VictoriaMetrics v1.85.2](./CHANGELOG.md#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** +**It is recommended upgrading to [VictoriaMetrics v1.85.2](https://docs.victoriametrics.com/changelog/#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** -* FEATURE: [vmalert](./vmalert.md): support `$for` or `.For` template variables in alert's annotations. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support `$for` or `.For` template variables in alert's annotations. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246). -* BUGFIX: [DataDog protocol parser](./README.md#how-to-send-data-from-datadog-agent): do not re-use `host` and `device` fields from the previously parsed messages if these fields are missing in the currently parsed message. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). -* BUGFIX: reduce CPU usage when the regex-based relabeling rules are applied to more than 100K unique Graphite metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466). The issue was introduced in [v1.82.0](./CHANGELOG.md#v1820). -* BUGFIX: do not block [merges](./README.md#storage) of small parts by merges of big parts on hosts with small number of CPU cores. This issue could result in the increasing number of `storage/small` parts while big merge is in progress. This, in turn, could result in increased CPU usage and memory usage during querying, since queries need to inspect bigger number of small parts. The issue has been introduced in [v1.85.0](./CHANGELOG.md#v1850). -* BUGFIX: [vmbackup](./vmbackup.md): fix the `The source request body for synchronous copy is too large and exceeds the maximum permissible limit (256MB)` error when performing backups to Azure blob storage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3477). +* BUGFIX: [DataDog protocol parser](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent): do not re-use `host` and `device` fields from the previously parsed messages if these fields are missing in the currently parsed message. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). +* BUGFIX: reduce CPU usage when the regex-based relabeling rules are applied to more than 100K unique Graphite metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466). The issue was introduced in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). +* BUGFIX: do not block [merges](https://docs.victoriametrics.com/#storage) of small parts by merges of big parts on hosts with small number of CPU cores. This issue could result in the increasing number of `storage/small` parts while big merge is in progress. This, in turn, could result in increased CPU usage and memory usage during querying, since queries need to inspect bigger number of small parts. The issue has been introduced in [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix the `The source request body for synchronous copy is too large and exceeds the maximum permissible limit (256MB)` error when performing backups to Azure blob storage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3477). ## [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0) Released at 2022-12-11 -**It is recommended upgrading to [VictoriaMetrics v1.85.2](./CHANGELOG.md#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** +**It is recommended upgrading to [VictoriaMetrics v1.85.2](https://docs.victoriametrics.com/changelog/#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** **Update note 1:** this release drops support for direct upgrade from VictoriaMetrics versions prior [v1.28.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.28.0). Please upgrade to `v1.84.0`, wait until `finished round 2 of background conversion` line is emitted to log by single-node VictoriaMetrics or by `vmstorage`, and then upgrade to newer releases. -**Update note 2:** this release splits `type="indexdb"` metrics into `type="indexdb/inmemory"` and `type="indexdb/file"` metrics. This may break old dashboards and alerting rules, which contain [label filter](./keyConcepts.md#filtering) on `{type="indexdb"}`. Such label filter must be substituted with `{type=~"indexdb.*"}`, so it matches `indexdb` from the previous releases and `indexdb/inmemory` + `indexdb/file` from new releases. It is recommended upgrading to the latest available dashboards and alerting rules mentioned in [these docs](./README.md#monitoring), since they already contain fixed label filters. +**Update note 2:** this release splits `type="indexdb"` metrics into `type="indexdb/inmemory"` and `type="indexdb/file"` metrics. This may break old dashboards and alerting rules, which contain [label filter](https://docs.victoriametrics.com/keyconcepts/#filtering) on `{type="indexdb"}`. Such label filter must be substituted with `{type=~"indexdb.*"}`, so it matches `indexdb` from the previous releases and `indexdb/inmemory` + `indexdb/file` from new releases. It is recommended upgrading to the latest available dashboards and alerting rules mentioned in [these docs](https://docs.victoriametrics.com/#monitoring), since they already contain fixed label filters. -**Update note 3:** this release deprecates `relabel_debug` and `metric_relabel_debug` config options in [scrape_configs](./sd_configs.md#scrape_configs). The `-relabelDebug`, `-remoteWrite.relabelDebug` and `-remoteWrite.urlRelabelDebug` command-line options are also deprecated. Use more powerful target-level relabel debugging and metric-level relabel debugging instead as documented [here](./vmagent.md#relabel-debug). +**Update note 3:** this release deprecates `relabel_debug` and `metric_relabel_debug` config options in [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs). The `-relabelDebug`, `-remoteWrite.relabelDebug` and `-remoteWrite.urlRelabelDebug` command-line options are also deprecated. Use more powerful target-level relabel debugging and metric-level relabel debugging instead as documented [here](https://docs.victoriametrics.com/vmagent/#relabel-debug). -* FEATURE: [vmagent](./vmagent.md): provide enhanced target-level and metric-level relabel debugging. See [these docs](./vmagent.md#relabel-debug) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407). -* FEATURE: leave a sample with the biggest value for identical timestamps per each `-dedup.minScrapeInterval` discrete interval when the [deduplication](./README.md#deduplication) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): provide enhanced target-level and metric-level relabel debugging. See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407). +* FEATURE: leave a sample with the biggest value for identical timestamps per each `-dedup.minScrapeInterval` discrete interval when the [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333). * FEATURE: add `-inmemoryDataFlushInterval` command-line flag, which can be used for controlling the frequency of in-memory data flush to disk. The data flush frequency can be reduced when VictoriaMetrics stores data to low-end flash device with limited number of write cycles (for example, on Raspberry PI). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). -* FEATURE: expose additional metrics for `indexdb` and `storage` parts stored in memory and for `indexdb` parts stored in files (see [storage docs](./README.md#storage) for technical details): +* FEATURE: expose additional metrics for `indexdb` and `storage` parts stored in memory and for `indexdb` parts stored in files (see [storage docs](https://docs.victoriametrics.com/#storage) for technical details): * `vm_active_merges{type="storage/inmemory"}` - active merges for in-memory `storage` parts * `vm_active_merges{type="indexdb/inmemory"}` - active merges for in-memory `indexdb` parts * `vm_active_merges{type="indexdb/file"}` - active merges for file-based `indexdb` parts @@ -97,99 +97,99 @@ Released at 2022-12-11 * `vm_rows{type="storage/inmemory"}` - the total number of in-memory `storage` rows * `vm_rows{type="indexdb/inmemory"}` - the total number of in-memory `indexdb` rows * `vm_rows{type="indexdb/file"}` - the total number of file-based `indexdb` rows -* FEATURE: [DataDog parser](./README.md#how-to-send-data-from-datadog-agent): add `device` tag when it is passed in the `device` field is present in the `series` object of the input request. Thanks to @PerGon for the provided [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3431). -* FEATURE: [vmagent](./vmagent.md): improve [service discovery](./sd_configs.md) performance when discovering big number of targets (10K and more). -* FEATURE: [vmagent](./vmagent.md): allow using `series_limit` option for [limiting the number of series a single scrape target generates](./vmagent.md#cardinality-limiter) in [stream parsing mode](./vmagent.md#stream-parsing-mode). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458). -* FEATURE: [vmagent](./vmagent.md): allow using `sample_limit` option for limiting the number of metrics a single scrape target can expose in every response sent over [stream parsing mode](./vmagent.md#stream-parsing-mode). -* FEATURE: [vmagent](./vmagent.md): add `exported_` prefix to metric names exported by scrape targets if these metric names clash with [automatically generated metrics](./vmagent.md#automatically-generated-metrics) such as `up`, `scrape_samples_scraped`, etc. This prevents from corruption of automatically generated metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406). -* FEATURE: [vmagent](./vmagent.md): make the `host` label optional in [DataDog data ingestion protocol](./README.md#how-to-send-data-from-datadog-agent). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): improve error message when the requested path cannot be properly parsed, so users could identify the issue and properly fix the path. Now the error message links to [url format docs](./Cluster-VictoriaMetrics.md#url-format). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402). -* FEATURE: [VictoriaMetrics enterprise cluster](./enterprise.md): add `-storageNode.discoveryInterval` command-line flag to `vmselect` and `vminsert` to control load on DNS servers when [automatic discovery of vmstorage nodes](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3417). -* FEATURE: [VictoriaMetrics enterprise cluster](./enterprise.md): allow reading and updating the list of `vmstorage` nodes at `vmselect` and `vminsert` nodes via file. See [automatic discovery of vmstorage](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery) for details. -* FEATURE: [vmalert](./vmalert.md): reduce memory and CPU usage by up to 50% on setups with thousands of recording/alerting groups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3464). -* FEATURE: [vmalert](./vmalert.md): add `-remoteWrite.sendTimeout` command-line flag, which allows configuring timeout for sending data to `-remoteWrite.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3408). -* FEATURE: [vmctl](./vmctl.md): add ability to migrate data between VictoriaMetrics clusters with automatic tenants discovery. See [these docs](./vmctl.md#cluster-to-cluster-migration-mode) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2930). -* FEATURE: [vmctl](./vmctl.md): add ability to copy data from sources via Prometheus `remote_read` protocol. See [these docs](./vmctl.md#migrating-data-by-remote-read-protocol). The related issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3132) and [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1101). -* FEATURE: [vmui](./README.md#vmui): allow changing timezones for the requested data. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3075). -* FEATURE: [vmui](./README.md#vmui): provide fast path for hiding results for all the queries except the given one by clicking `eye` icon with `ctrl` key pressed. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3446). -* FEATURE: [MetricsQL](./MetricsQL.md): add `range_trim_spikes(phi, q)` function for trimming `phi` percent of the largest spikes per each time series returned by `q`. See [these docs](./MetricsQL.md#range_trim_spikes). -* FEATURE: [MetricsQL](./MetricsQL.md): allow passing `inf` arg into [limitk](./MetricsQL.md#limitk), [topk](./MetricsQL.md#topk), [bottomk](./MetricsQL.md) and other functions, which accept numeric arg, which limits the number of output time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461). -* FEATURE: [vmgateway](./vmgateway.md): add support for JWT token signature verification. See [these docs](./vmgateway.md#jwt-signature-verification) for details. -* FEATURE: put the version of VictoriaMetrics in the first message of [query trace](./README.md#query-tracing). This should simplify debugging. +* FEATURE: [DataDog parser](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent): add `device` tag when it is passed in the `device` field is present in the `series` object of the input request. Thanks to @PerGon for the provided [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3431). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): improve [service discovery](https://docs.victoriametrics.com/sd_configs/) performance when discovering big number of targets (10K and more). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow using `series_limit` option for [limiting the number of series a single scrape target generates](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow using `sample_limit` option for limiting the number of metrics a single scrape target can expose in every response sent over [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `exported_` prefix to metric names exported by scrape targets if these metric names clash with [automatically generated metrics](https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics) such as `up`, `scrape_samples_scraped`, etc. This prevents from corruption of automatically generated metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): make the `host` label optional in [DataDog data ingestion protocol](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): improve error message when the requested path cannot be properly parsed, so users could identify the issue and properly fix the path. Now the error message links to [url format docs](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402). +* FEATURE: [VictoriaMetrics enterprise cluster](https://docs.victoriametrics.com/enterprise/): add `-storageNode.discoveryInterval` command-line flag to `vmselect` and `vminsert` to control load on DNS servers when [automatic discovery of vmstorage nodes](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3417). +* FEATURE: [VictoriaMetrics enterprise cluster](https://docs.victoriametrics.com/enterprise/): allow reading and updating the list of `vmstorage` nodes at `vmselect` and `vminsert` nodes via file. See [automatic discovery of vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): reduce memory and CPU usage by up to 50% on setups with thousands of recording/alerting groups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3464). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-remoteWrite.sendTimeout` command-line flag, which allows configuring timeout for sending data to `-remoteWrite.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3408). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add ability to migrate data between VictoriaMetrics clusters with automatic tenants discovery. See [these docs](https://docs.victoriametrics.com/vmctl/#cluster-to-cluster-migration-mode) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2930). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add ability to copy data from sources via Prometheus `remote_read` protocol. See [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol). The related issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3132) and [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1101). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): allow changing timezones for the requested data. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3075). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): provide fast path for hiding results for all the queries except the given one by clicking `eye` icon with `ctrl` key pressed. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3446). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `range_trim_spikes(phi, q)` function for trimming `phi` percent of the largest spikes per each time series returned by `q`. See [these docs](https://docs.victoriametrics.com/metricsql/#range_trim_spikes). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow passing `inf` arg into [limitk](https://docs.victoriametrics.com/metricsql/#limitk), [topk](https://docs.victoriametrics.com/metricsql/#topk), [bottomk](https://docs.victoriametrics.com/metricsql/) and other functions, which accept numeric arg, which limits the number of output time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461). +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add support for JWT token signature verification. See [these docs](https://docs.victoriametrics.com/vmgateway/#jwt-signature-verification) for details. +* FEATURE: put the version of VictoriaMetrics in the first message of [query trace](https://docs.victoriametrics.com/#query-tracing). This should simplify debugging. -* BUGFIX: [vmagent](./vmagent.md): fix the `The request did not have a subscription or a valid tenant level resource provider` error when discovering Azure targets with [azure_sd_configs](./sd_configs.md#azure_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247). -* BUGFIX: [vmalert](./vmalert.md): properly pass HTTP headers during the alert state restore procedure. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418). -* BUGFIX: [vmalert](./vmalert.md): properly specify rule evaluation step during the [replay mode](./vmalert.md#rules-backfilling). The `step` value was previously overriden by `-datasource.queryStep` command-line flag. -* BUGFIX: [vmalert](./vmalert.md): properly return the error message from remote-write failures. Before, error was ignored and only `vmalert_remotewrite_errors_total` was incremented. -* BUGFIX: [vmui](./README.md#vmui): fix sticky tooltip sizing, which could prevent from closing the tooltip. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427). -* BUGFIX: [vmui](./README.md#vmui): properly put multi-line queries in the url, so it could be copy-n-pasted and opened without issues in a new browser tab. Previously the url for multi-line query couldn't be opened. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444). -* BUGFIX: [vmui](./README.md#vmui): correctly handle `up` and `down` keypresses when editing multi-line queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix the `The request did not have a subscription or a valid tenant level resource provider` error when discovering Azure targets with [azure_sd_configs](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly pass HTTP headers during the alert state restore procedure. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly specify rule evaluation step during the [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling). The `step` value was previously overriden by `-datasource.queryStep` command-line flag. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return the error message from remote-write failures. Before, error was ignored and only `vmalert_remotewrite_errors_total` was incremented. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix sticky tooltip sizing, which could prevent from closing the tooltip. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly put multi-line queries in the url, so it could be copy-n-pasted and opened without issues in a new browser tab. Previously the url for multi-line query couldn't be opened. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): correctly handle `up` and `down` keypresses when editing multi-line queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445). ## [v1.84.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.84.0) Released at 2022-11-25 -**It is recommended upgrading to [VictoriaMetrics v1.85.2](./CHANGELOG.md#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** +**It is recommended upgrading to [VictoriaMetrics v1.85.2](https://docs.victoriametrics.com/changelog/#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** -* FEATURE: add support for [Pushgateway data import format](https://github.com/prometheus/pushgateway#url) via `/api/v1/import/prometheus` url. See [these docs](./README.md#how-to-import-data-in-prometheus-exposition-format) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415). Thanks to @PerGon for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3360). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `http://:8481/admin/tenants` API endpoint for returning a list of registered tenants. See [these docs](./Cluster-VictoriaMetrics.md#url-format) for details. -* FEATURE: [VictoriaMetrics enterprise](./enterprise.md): add `-storageNode.filter` command-line flag for filtering the [discovered vmstorage nodes](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery) with arbitrary regular expressions. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353). -* FEATURE: [MetricsQL](./MetricsQL.md): allow using numeric values with `K`, `Ki`, `M`, `Mi`, `G`, `Gi`, `T` and `Ti` suffixes inside MetricsQL queries. For example `8Ki` equals to `8*1024`, while `8.2M` equals to `8.2*1000*1000`. -* FEATURE: [MetricsQL](./MetricsQL.md): add [range_normalize](./MetricsQL.md#range_normalize) function for normalizing multiple time series into `[0...1]` value range. This function is useful for correlation analysis of time series with distinct value ranges. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3167). -* FEATURE: [MetricsQL](./MetricsQL.md): add [range_linear_regression](./MetricsQL.md#range_linear_regression) function for calculating [simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression) over the input time series on the selected time range. This function is useful for predictions and capacity planning. For example, `range_linear_regression(process_resident_memory_bytes)` can predict future memory usage based on the past memory usage. -* FEATURE: [MetricsQL](./MetricsQL.md): add [range_stddev](./MetricsQL.md#range_stddev) and [range_stdvar](./MetricsQL.md#range_stdvar) functions. -* FEATURE: [MetricsQL](./MetricsQL.md): optimize `expr1 op expr2` query when `expr1` returns an empty result. In this case there is no sense in executing `expr2` for `op` not equal to `or`, since the end result will be empty according to [PromQL series matching rules](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349). Thanks to @jianglinjian for pointing to this case. -* FEATURE: [vmui](./README.md#vmui): add the ability to upload/paste JSON to investigate the trace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3308) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3310). -* FEATURE: [vmui](./README.md#vmui): reduce JS bundle size from 200Kb to 100Kb. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3298). -* FEATURE: [vmui](./README.md#vmui): add the ability to hide results of a particular query by clicking the `eye` icon. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3359). -* FEATURE: [vmui](./README.md#vmui): add copy button to row on Table view. The button copies row in MetricQL format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2815). -* FEATURE: [vmui](./README.md#vmui): add compact table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3241). -* FEATURE: [vmui](./README.md#vmui): add the ability to "stick" a tooltip on the chart by clicking on a data point. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3321) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3376) -* FEATURE: [vmui](./README.md#vmui): add the ability to set up series custom limits. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3297). -* FEATURE: [vmalert](./vmalert.md): add default alert list for vmalert's metrics. See [alerts-vmalert.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmalert.yml). -* FEATURE: [vmagent](./vmagent.md): expose `vmagent_relabel_config_*`, `vm_relabel_config_*` and `vm_promscrape_config_*` metrics for tracking relabel and scrape configuration hot-reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345). +* FEATURE: add support for [Pushgateway data import format](https://github.com/prometheus/pushgateway#url) via `/api/v1/import/prometheus` url. See [these docs](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415). Thanks to @PerGon for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3360). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `http://:8481/admin/tenants` API endpoint for returning a list of registered tenants. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format) for details. +* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): add `-storageNode.filter` command-line flag for filtering the [discovered vmstorage nodes](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) with arbitrary regular expressions. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow using numeric values with `K`, `Ki`, `M`, `Mi`, `G`, `Gi`, `T` and `Ti` suffixes inside MetricsQL queries. For example `8Ki` equals to `8*1024`, while `8.2M` equals to `8.2*1000*1000`. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [range_normalize](https://docs.victoriametrics.com/metricsql/#range_normalize) function for normalizing multiple time series into `[0...1]` value range. This function is useful for correlation analysis of time series with distinct value ranges. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3167). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [range_linear_regression](https://docs.victoriametrics.com/metricsql/#range_linear_regression) function for calculating [simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression) over the input time series on the selected time range. This function is useful for predictions and capacity planning. For example, `range_linear_regression(process_resident_memory_bytes)` can predict future memory usage based on the past memory usage. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [range_stddev](https://docs.victoriametrics.com/metricsql/#range_stddev) and [range_stdvar](https://docs.victoriametrics.com/metricsql/#range_stdvar) functions. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): optimize `expr1 op expr2` query when `expr1` returns an empty result. In this case there is no sense in executing `expr2` for `op` not equal to `or`, since the end result will be empty according to [PromQL series matching rules](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349). Thanks to @jianglinjian for pointing to this case. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to upload/paste JSON to investigate the trace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3308) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3310). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): reduce JS bundle size from 200Kb to 100Kb. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3298). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to hide results of a particular query by clicking the `eye` icon. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3359). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add copy button to row on Table view. The button copies row in MetricQL format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2815). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add compact table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3241). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to "stick" a tooltip on the chart by clicking on a data point. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3321) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3376) +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to set up series custom limits. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3297). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add default alert list for vmalert's metrics. See [alerts-vmalert.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmalert.yml). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `vmagent_relabel_config_*`, `vm_relabel_config_*` and `vm_promscrape_config_*` metrics for tracking relabel and scrape configuration hot-reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly return an empty result from [limit_offset](./MetricsQL.md#limit_offset) if the `offset` arg exceeds the number of inner time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312). -* BUGFIX: [vmagent](./vmagent.md): properly discover GCE zones when `filter` option is set at [gce_sd_configs](./sd_configs.md#gce_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202). -* BUGFIX: [vmui](./README.md#vmui): properly display the requested graph on the requested time range when navigating from Prometheus URL in Grafana. -* BUGFIX: [vmui](./README.md#vmui): properly display wide tables. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3153). -* BUGFIX: reduce CPU usage spikes and memory usage spikes under high data ingestion rate introduced in [v1.83.0](./CHANGELOG.md#v1830). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly return an empty result from [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) if the `offset` arg exceeds the number of inner time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover GCE zones when `filter` option is set at [gce_sd_configs](https://docs.victoriametrics.com/sd_configs/#gce_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly display the requested graph on the requested time range when navigating from Prometheus URL in Grafana. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly display wide tables. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3153). +* BUGFIX: reduce CPU usage spikes and memory usage spikes under high data ingestion rate introduced in [v1.83.0](https://docs.victoriametrics.com/changelog/#v1830). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343). ## [v1.83.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.83.1) Released at 2022-11-10 -**It is recommended upgrading to [VictoriaMetrics v1.85.2](./CHANGELOG.md#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** +**It is recommended upgrading to [VictoriaMetrics v1.85.2](https://docs.victoriametrics.com/changelog/#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** -* FEATURE: [vmagent](./vmagent.md): expose `__meta_consul_partition` label for targets discovered via [consul_sd_configs](./sd_configs.md#consul_sd_configs) in the same way as [Prometheus 2.40 does](https://github.com/prometheus/prometheus/pull/11482). -* FEATURE: [vmui](./README.md#vmui): show the [query trace](./README.md#query-tracing) in JSON view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2814). Thanks to @michal-kralik for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3316). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `__meta_consul_partition` label for targets discovered via [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs) in the same way as [Prometheus 2.40 does](https://github.com/prometheus/prometheus/pull/11482). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show the [query trace](https://docs.victoriametrics.com/#query-tracing) in JSON view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2814). Thanks to @michal-kralik for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3316). -* BUGFIX: [VictoriaMetrics enterprise](./enterprise.md): fix a panic at `vminsert` when the discovered list of `vmstorage` nodes is changed during [automatic vmstorage discovery](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329). +* BUGFIX: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): fix a panic at `vminsert` when the discovered list of `vmstorage` nodes is changed during [automatic vmstorage discovery](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329). * BUGFIX: properly register new time series in per-day inverted index if they were ingested during the last 10 seconds of the day. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3309). Thanks to @lmarszal for the bugreport and for the [initial fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3320). -* BUGFIX: reduce the increased memory usage spikes for some workloads. The issue was introduced in [v1.83.0](./CHANGELOG.md#v1830). -* BUGFIX: properly accept [OpenTSDB telnet put lines](./README.md#sending-data-via-telnet-put-protocol) without tags without the need to specify the trailing whitespace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290). +* BUGFIX: reduce the increased memory usage spikes for some workloads. The issue was introduced in [v1.83.0](https://docs.victoriametrics.com/changelog/#v1830). +* BUGFIX: properly accept [OpenTSDB telnet put lines](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol) without tags without the need to specify the trailing whitespace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290). ## [v1.83.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.83.0) Released at 2022-10-29 -**It is recommended upgrading to [VictoriaMetrics v1.85.2](./CHANGELOG.md#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** +**It is recommended upgrading to [VictoriaMetrics v1.85.2](https://docs.victoriametrics.com/changelog/#v1852) because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502), which may result in incomplete query results for historical time series.** -**Update note 1:** the `indexdb/tagFilters` cache type at [/metrics](./README.md#monitoring) has been renamed to `indexdb/tagFiltersToMetricIDs` in order to make its purpose more clear. +**Update note 1:** the `indexdb/tagFilters` cache type at [/metrics](https://docs.victoriametrics.com/#monitoring) has been renamed to `indexdb/tagFiltersToMetricIDs` in order to make its purpose more clear. -**Update note 2:** [vmalert](./vmalert.md): the `crlfEscape` [template function](./vmalert.md#template-functions) becomes obsolete starting from this release. It can be safely removed from alerting templates, since `\n` chars are properly escaped with other `*Escape` functions now. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue for details. +**Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert/): the `crlfEscape` [template function](https://docs.victoriametrics.com/vmalert/#template-functions) becomes obsolete starting from this release. It can be safely removed from alerting templates, since `\n` chars are properly escaped with other `*Escape` functions now. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue for details. -* FEATURE: [VictoriaMetrics enterprise](./enterprise.md): add support for automatic `vmstorage` nodes discovering and updating at `vmselect` and `vminsert`. See [these docs](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery). -* FEATURE: [VictoriaMetrics enterprise](./enterprise.md): allow configuring multiple retentions for distinct sets of time series. See [these docs](./README.md#retention-filters), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request. -* FEATURE: [VictoriaMetric cluster enterprise](./enterprise.md): add support for multiple retentions for distinct tenants - see [these docs](./Cluster-VictoriaMetrics.md#retention-filters) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request. +* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): add support for automatic `vmstorage` nodes discovering and updating at `vmselect` and `vminsert`. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery). +* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): allow configuring multiple retentions for distinct sets of time series. See [these docs](https://docs.victoriametrics.com/#retention-filters), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request. +* FEATURE: [VictoriaMetric cluster enterprise](https://docs.victoriametrics.com/enterprise/): add support for multiple retentions for distinct tenants - see [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#retention-filters) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289) feature request. * FEATURE: allow limiting memory usage on a per-query basis with `-search.maxMemoryPerQuery` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203). -* FEATURE: allow referring environment variables inside command-line flags via `%{ENV_VAR}` syntax. For example, if `AUTH_KEY=top-secret` environment variable is set, then `-metricsAuthKey=%{AUTH_KEY}` command-line flag is automatically expanded to `-storageDataPath=top-secret` at VictoriaMetrics startup. See [these docs](./README.md#environment-variables) for details. +* FEATURE: allow referring environment variables inside command-line flags via `%{ENV_VAR}` syntax. For example, if `AUTH_KEY=top-secret` environment variable is set, then `-metricsAuthKey=%{AUTH_KEY}` command-line flag is automatically expanded to `-storageDataPath=top-secret` at VictoriaMetrics startup. See [these docs](https://docs.victoriametrics.com/#environment-variables) for details. * FEATURE: allow referring environment variables inside other environment variables via `%{ENV_VAR}` syntax. For example, if `A=a-%{B}`, `B=b-%{C}` and `C=c` env vars are set, then VictoriaMetrics components automatically expand them to `A=a-b-c`, `B=b-c` and `C=c` on startup. -* FEATURE: [vmagent](./vmagent.md): drop all the labels with `__` prefix from discovered targets in the same way as Prometheus does according to [this article](https://www.robustperception.io/life-of-a-label/). Previously the following labels were available during [metric-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs): `__address__`, `__scheme__`, `__metrics_path__`, `__scrape_interval__`, `__scrape_timeout__`, `__param_*`. Now these labels are available only during [target-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). This should reduce CPU usage and memory usage for `vmagent` setups, which scrape big number of targets. -* FEATURE: [vmagent](./vmagent.md): improve the performance for metric-level [relabeling](./vmagent.md#relabeling), which can be applied via `metric_relabel_configs` section at [scrape_configs](./sd_configs.md#scrape_configs), via `-remoteWrite.relabelConfig` or via `-remoteWrite.urlRelabelConfig` command-line options. -* FEATURE: [vmagent](./vmagent.md): allow specifying full url in scrape target addresses (aka `__address__` label). This makes valid the following `-promscrape.config`: +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): drop all the labels with `__` prefix from discovered targets in the same way as Prometheus does according to [this article](https://www.robustperception.io/life-of-a-label/). Previously the following labels were available during [metric-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs): `__address__`, `__scheme__`, `__metrics_path__`, `__scrape_interval__`, `__scrape_timeout__`, `__param_*`. Now these labels are available only during [target-level relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). This should reduce CPU usage and memory usage for `vmagent` setups, which scrape big number of targets. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): improve the performance for metric-level [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling), which can be applied via `metric_relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs), via `-remoteWrite.relabelConfig` or via `-remoteWrite.urlRelabelConfig` command-line options. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying full url in scrape target addresses (aka `__address__` label). This makes valid the following `-promscrape.config`: ```yaml scrape_configs: @@ -208,205 +208,205 @@ Released at 2022-10-29 See [the corresponding issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3208). -* FEATURE: [vmagent](./vmagent.md): allow controlling [staleness tracking](./vmagent.md#prometheus-staleness-markers) on a per-[scrape_config](./sd_configs.md#scrape_configs) basis by specifying `no_stale_markers: true` or `no_stale_markers: false` option in the corresponding [scrape_config](./sd_configs.md#scrape_configs). -* FEATURE: [vmalert](./vmalert.md): add `strvalue` and `stripDomain` [template functions](./vmalert.md#template-functions) in order to improve compatibility with Prometheus. -* FEATURE: [vmalert](./vmalert.md): add `jsonEscape` and `htmlEscape` [template functions](./vmalert.md#template-functions). -* FEATURE: [vmui](./README.md#vmui): limit the number of plotted series. This should prevent from browser crashes or hangs when the query returns big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3155). -* FEATURE: [vmui](./README.md#vmui): reduce memory usage when querying big number of time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240). -* FEATURE: [vmui](./README.md#vmui): add responsive styles for small screens. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3239) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3256). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow controlling [staleness tracking](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) on a per-[scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs) basis by specifying `no_stale_markers: true` or `no_stale_markers: false` option in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `strvalue` and `stripDomain` [template functions](https://docs.victoriametrics.com/vmalert/#template-functions) in order to improve compatibility with Prometheus. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `jsonEscape` and `htmlEscape` [template functions](https://docs.victoriametrics.com/vmalert/#template-functions). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): limit the number of plotted series. This should prevent from browser crashes or hangs when the query returns big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3155). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): reduce memory usage when querying big number of time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add responsive styles for small screens. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3239) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3256). * FEATURE: log error if some environment variables referred at `-promscrape.config` via `%{ENV_VAR}` aren't found. This should prevent from silent using incorrect config files. * FEATURE: immediately shut down VictoriaMetrics apps on the second SIGINT or SIGTERM signal if they couldn't be finished gracefully for some reason after receiving the first signal. -* FEATURE: improve the performance of [/api/v1/series](./url-examples.md#apiv1series) endpoint by eliminating loading of unused `TSID` data during the API call. -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): add functionality for automated restore from backup. See [these docs](./vmbackupmanager.md#how-to-restore-backup-via-cli). +* FEATURE: improve the performance of [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) endpoint by eliminating loading of unused `TSID` data during the API call. +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): add functionality for automated restore from backup. See [these docs](https://docs.victoriametrics.com/vmbackupmanager/#how-to-restore-backup-via-cli). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](./MetricsQL.md#histogram_quantile) and [histogram_share](./MetricsQL.md#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225). -* BUGFIX: [vmalert](./vmalert.md): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level. -* BUGFIX: [vmalert](./vmalert.md): fix panic if `vmalert` runs with `-clusterMode` command-line flag in [multitenant mode](./vmalert.md#multitenancy). The issue has been introduced in [v1.82.0](./CHANGELOG.md#v1820). -* BUGFIX: [vmalert](./vmalert.md): properly escape string passed to `quotesEscape` [template function](./vmalert.md#template-functions), so it can be safely embedded into JSON string. This makes obsolete the `crlfEscape` function. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue. -* BUGFIX: [vmagent](./vmagent.md): do not show invalid error message in Kubernetes service discovery: `cannot parse WatchEvent json response: EOF`. The invalid error message has been appeared in [v1.82.0](./CHANGELOG.md#v1820). -* BUGFIX: [vmagent](./vmagent.md): properly add `exported_` prefix to metric labels, which clashing with scrape target labels if `honor_labels: true` option isn't set in [scrape_config](./sd_configs.md#scrape_configs). Previously some `exported_` prefixes were missing in the resulting metric labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278). The issue has been introduced in [v1.82.0](./CHANGELOG.md#v1820). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) and [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix panic if `vmalert` runs with `-clusterMode` command-line flag in [multitenant mode](https://docs.victoriametrics.com/vmalert/#multitenancy). The issue has been introduced in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly escape string passed to `quotesEscape` [template function](https://docs.victoriametrics.com/vmalert/#template-functions), so it can be safely embedded into JSON string. This makes obsolete the `crlfEscape` function. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not show invalid error message in Kubernetes service discovery: `cannot parse WatchEvent json response: EOF`. The invalid error message has been appeared in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly add `exported_` prefix to metric labels, which clashing with scrape target labels if `honor_labels: true` option isn't set in [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). Previously some `exported_` prefixes were missing in the resulting metric labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278). The issue has been introduced in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). * BUGFIX: `vmselect`: expose missing metric `vm_cache_size_max_bytes{type="promql/rollupResult"}` . This metric is used for monitoring rollup cache usage with the query `vm_cache_size_bytes{type="promql/rollupResult"} / vm_cache_size_max_bytes{type="promql/rollupResult"}` in the same way as this is done for other cache types. ## [v1.82.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.1) Released at 2022-10-14 -* BUGFIX: [vmui](./README.md#vmui): automatically update graph, legend and url after the removal of query field. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205). -* BUGFIX: [vmalert](./vmalert.md): remove duplicate `alertname` JSON entry from generated alerts. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3053). Thanks to @Howie59 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3182)! -* BUGFIX: [vmalert](./vmalert.md): fix integration with Grafana via `-vmalert.proxyURL`, which has been broken in [v1.82.0](./CHANGELOG.md#v1820). See [this issue](https://github.com/VictoriaMetrics/helm-charts/issues/391). -* BUGFIX: [vmbackup](./vmbackup.md): set default region to `us-east-1` if `AWS_REGION` environment variable isn't set. The issue was introduced in [vmbackup v1.82.0](./CHANGELOG.md#v1820). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3224). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix deletion of old backups at [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs/). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly apply regex filters when searching for time series. Previously unexpected time series could be returned from regex filter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227). The issue was introduced in [v1.82.0](./CHANGELOG.md#v1820). -* BUGFIX: [vmagent](./vmagent.md): properly apply `if` section with regex filters. Previously unexpected metrics could be returned from `if` section. The issue was introduced in [v1.82.0](./CHANGELOG.md#v1820). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): automatically update graph, legend and url after the removal of query field. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): remove duplicate `alertname` JSON entry from generated alerts. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3053). Thanks to @Howie59 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3182)! +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix integration with Grafana via `-vmalert.proxyURL`, which has been broken in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). See [this issue](https://github.com/VictoriaMetrics/helm-charts/issues/391). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): set default region to `us-east-1` if `AWS_REGION` environment variable isn't set. The issue was introduced in [vmbackup v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3224). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix deletion of old backups at [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs/). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly apply regex filters when searching for time series. Previously unexpected time series could be returned from regex filter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227). The issue was introduced in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply `if` section with regex filters. Previously unexpected metrics could be returned from `if` section. The issue was introduced in [v1.82.0](https://docs.victoriametrics.com/changelog/#v1820). ## [v1.82.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.82.0) Released at 2022-10-07 -**It isn't recommended to use VictoriaMetrics and vmagent v1.82.0 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227), which may result in incorrect query results and [relabeling](./vmagent.md#relabeling) results. Upgrade to [v1.82.1](./CHANGELOG.md#v1821) instead.** +**It isn't recommended to use VictoriaMetrics and vmagent v1.82.0 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227), which may result in incorrect query results and [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) results. Upgrade to [v1.82.1](https://docs.victoriametrics.com/changelog/#v1821) instead.** -**Update note 1:** this release changes data format for [/api/v1/export/native](./README.md#how-to-export-data-in-native-format) in incompatible way, so it cannot be imported into older version of VictoriaMetrics via [/api/v1/import/native](./README.md#how-to-import-data-in-native-format). +**Update note 1:** this release changes data format for [/api/v1/export/native](https://docs.victoriametrics.com/#how-to-export-data-in-native-format) in incompatible way, so it cannot be imported into older version of VictoriaMetrics via [/api/v1/import/native](https://docs.victoriametrics.com/#how-to-import-data-in-native-format). -**Update note 2:** [vmalert](./vmalert.md) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. +**Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert/) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. **Update note 3:** `vm_account_id` and `vm_project_id` labels must be passed to tcp-based `Graphite`, `InfluxDB` and `OpenTSDB` endpoints -at [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) instead of undocumented +at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) instead of undocumented `VictoriaMetrics_AccountID` and `VictoriaMetrics_ProjectID` labels when writing samples to the needed tenant. -See [these docs](./Cluster-VictoriaMetrics.md#multitenancy-via-labels) for details. +See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) for details. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): support specifying tenant ids via `vm_account_id` and `vm_project_id` labels. See [these docs](./Cluster-VictoriaMetrics.md#multitenancy-via-labels) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970). -* FEATURE: [vmagent](./vmagent.md): improve [relabeling](./vmagent.md#relabeling) performance by up to 3x for non-trivial `regex` values such as `([^:]+):.+`, which can be used for extracting a `host` part from `host:port` label value. -* FEATURE: [MetricsQL](./MetricsQL.md): improve performance by up to 4x for queries containing non-trivial `regex` filters such as `{path=~"/foo/.+|/bar"}`. -* FEATURE: improve performance scalability on systems with many CPU cores for [/federate](./README.md#federation) and [/api/v1/export/...](./README.md#how-to-export-time-series) endpoints. -* FEATURE: sanitize metric names for data ingested via [DataDog protocol](./README.md#how-to-send-data-from-datadog-agent) according to [DataDog metric naming](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics). The behaviour can be disabled by passing `-datadog.sanitizeMetricName=false` command-line flag. Thanks to @PerGon for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105). -* FEATURE: add `-usePromCompatibleNaming` command-line flag to [vmagent](./vmagent.md), to single-node VictoriaMetrics and to `vminsert` component of VictoriaMetrics cluster. This flag can be used for normalizing the ingested metric names and label names to [Prometheus-compatible form](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). If this flag is set, then all the chars unsupported by Prometheus are replaced with `_` chars in metric names and labels of the ingested samples. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113). -* FEATURE: accept whitespace in metric names and tags ingested via [Graphite plaintext protocol](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) according to [the specs](https://graphite.readthedocs.io/en/latest/tags.html). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): support specifying tenant ids via `vm_account_id` and `vm_project_id` labels. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): improve [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) performance by up to 3x for non-trivial `regex` values such as `([^:]+):.+`, which can be used for extracting a `host` part from `host:port` label value. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): improve performance by up to 4x for queries containing non-trivial `regex` filters such as `{path=~"/foo/.+|/bar"}`. +* FEATURE: improve performance scalability on systems with many CPU cores for [/federate](https://docs.victoriametrics.com/#federation) and [/api/v1/export/...](https://docs.victoriametrics.com/#how-to-export-time-series) endpoints. +* FEATURE: sanitize metric names for data ingested via [DataDog protocol](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) according to [DataDog metric naming](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics). The behaviour can be disabled by passing `-datadog.sanitizeMetricName=false` command-line flag. Thanks to @PerGon for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105). +* FEATURE: add `-usePromCompatibleNaming` command-line flag to [vmagent](https://docs.victoriametrics.com/vmagent/), to single-node VictoriaMetrics and to `vminsert` component of VictoriaMetrics cluster. This flag can be used for normalizing the ingested metric names and label names to [Prometheus-compatible form](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). If this flag is set, then all the chars unsupported by Prometheus are replaced with `_` chars in metric names and labels of the ingested samples. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113). +* FEATURE: accept whitespace in metric names and tags ingested via [Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) according to [the specs](https://graphite.readthedocs.io/en/latest/tags.html). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102). * FEATURE: check the correctess of raw sample timestamps stored on disk when reading them. This reduces the probability of possible silent corruption of the data stored on disk. This should help [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011). -* FEATURE: atomically delete directories with snapshots, parts and partitions at [storage level](./README.md#storage). Previously such directories can be left in partially deleted state when the deletion operation was interrupted by unclean shutdown. This may result in `cannot open file ...: no such file or directory` error on the next start. The probability of this error was quite high when NFS or EFS was used as persistent storage for VictoriaMetrics data. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038). -* FEATURE: set the `start` arg to `end - 5 minutes` if isn't passed explicitly to [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052). +* FEATURE: atomically delete directories with snapshots, parts and partitions at [storage level](https://docs.victoriametrics.com/#storage). Previously such directories can be left in partially deleted state when the deletion operation was interrupted by unclean shutdown. This may result in `cannot open file ...: no such file or directory` error on the next start. The probability of this error was quite high when NFS or EFS was used as persistent storage for VictoriaMetrics data. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038). +* FEATURE: set the `start` arg to `end - 5 minutes` if isn't passed explicitly to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052). * FEATURE: allow to define the minimum TLS version to use when accepting https requests to VictoriaMetrics components if `-tls` command-line flag is set. The minimum TLS version can be set via `-tlsMinVersion` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3090). -* FEATURE: [vmctl](./vmctl.md): add `vm-native-step-interval` command line flag for `vm-native` mode. New option allows splitting the import process into chunks by time interval. This helps migrating data sets with high churn rate and provides better control over the process. See [feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733). -* FEATURE: [vmui](./README.md#vmui): add `top queries` tab, which shows various stats for recently executed queries. See [these docs](./README.md#top-queries) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2707). -* FEATURE: [vmui](./README.md#vmui): move the "Execute Query" and "Add Query" buttons below the query fields, change icon for remove query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3101). -* FEATURE: [vmui](./README.md#vmui): set the maximum number of queries to 4, remove multi Y-axes, left one for all queries and dotted lines to indicate queries in the graph. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169). -* FEATURE: [vmalert](./vmalert.md): add `debug` mode to the alerting rule settings for printing additional information into logs during evaluation. See `debug` param in [alerting rule config](./vmalert.md#alerting-rules). -* FEATURE: [vmalert](./vmalert.md): add experimental feature for displaying last 10 states of the rule (recording or alerting) evaluation. The state is available on the Rule page, which can be opened by clicking on `Details` link next to Rule's name on the `/groups` page. -* FEATURE: [vmalert](./vmalert.md): allow using extra labels in annotations. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013). -* FEATURE: [vmalert](./vmalert.md): allow configuring authorization params per list of targets in vmalert's notifier config for `static_configs`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690). -* FEATURE: [vmalert](./vmalert.md): allow using `{{$labels}}` for templating in command-line flag `-external.alert.source`. The change supposed to provide additional flexibility for generating alert's source link based on labels values. -* FEATURE: [vmalert](./vmalert.md): add `vm_account_id` and `vm_project_id` labels to results of alerting and recording rules if `-clusterMode` is enabled. This improves [multitenant support in vmalert](./vmalert.md#multitenancy). -* FEATURE: [vmagent](./vmagent.md): minimize the time needed for reading large responses from scrape targets in [stream parsing mode](./vmagent.md#stream-parsing-mode). This should reduce scrape durations for such targets as [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) running in a big Kubernetes cluster. -* FEATURE: [MetricsQL](./MetricsQL.md): add [sort_by_label_numeric](./MetricsQL.md#sort_by_label_numeric) and [sort_by_label_numeric_desc](./MetricsQL.md#sort_by_label_numeric_desc) functions for [numeric sort](https://www.gnu.org/software/coreutils/manual/html_node/Version-sort-is-not-the-same-as-numeric-sort.html) of input time series by the specified labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2938). -* FEATURE: [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md): retry GCS operations for up to 3 minutes on temporary failures. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147). -* FEATURE: [vmbackup](./vmbackup.md): add support for saving / restoring backups to / from [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029). -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): expose `vm_backup_in_flight` metric, which can be used for determining which backup types - latest, hourly, daily, weekly or monthly - are currently executed. -* FEATURE: [vmgateway](./vmgateway.md): add ability to extract JWT authorization token from non-standard HTTP header by passing it via `-auth.httpHeader` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3054). -* FEATURE: [vmagent](./vmagent.md): expose `__meta_ec2_region` label for [ec2_sd_config](./sd_configs.md#ec2_sd_configs) in the same way as [Prometheus 2.39 does](https://github.com/prometheus/prometheus/pull/11326). -* FEATURE: [vmagent](./vmagent.md): accept data ingestion requests via paths starting from `/prometheus` prefix in the same way as [VictoriaMetrics does](./README.md#how-to-import-time-series-data). For example, `vmagent` now accepts Prometheus `remote_write` data via both `/api/v1/write` and `/prometheus/api/v1/write`. This simplifies switching between single-node VictoriaMetrics and `vmagent`. -* FEATURE: [vmagent](./vmagent.md): add `external_labels` from `global` section at `-promscrape.config` after the [relabeling](./vmagent.md#relabeling) is applied to scraped metrics. This aligns with Prometheus behaviour. Previously the `external_labels` were added to scrape targets, so they could be modified during relabeling. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137). -* FEATURE: [vmagent](./vmagent.md): allow specifying per-`-remoteWrite.url` limits for on-disk size for pending data via `-remoteWrite.maxDiskUsagePerURL` command-line flag. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): log clear error when multiple identical `-storageNode` command-line flags are passed to `vmselect` or to `vminsert`. Previously these components were crashed with cryptic panic `metric ... is already registered` in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3076). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `vm-native-step-interval` command line flag for `vm-native` mode. New option allows splitting the import process into chunks by time interval. This helps migrating data sets with high churn rate and provides better control over the process. See [feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `top queries` tab, which shows various stats for recently executed queries. See [these docs](https://docs.victoriametrics.com/#top-queries) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2707). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): move the "Execute Query" and "Add Query" buttons below the query fields, change icon for remove query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3101). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): set the maximum number of queries to 4, remove multi Y-axes, left one for all queries and dotted lines to indicate queries in the graph. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `debug` mode to the alerting rule settings for printing additional information into logs during evaluation. See `debug` param in [alerting rule config](https://docs.victoriametrics.com/vmalert/#alerting-rules). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add experimental feature for displaying last 10 states of the rule (recording or alerting) evaluation. The state is available on the Rule page, which can be opened by clicking on `Details` link next to Rule's name on the `/groups` page. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow using extra labels in annotations. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow configuring authorization params per list of targets in vmalert's notifier config for `static_configs`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow using `{{$labels}}` for templating in command-line flag `-external.alert.source`. The change supposed to provide additional flexibility for generating alert's source link based on labels values. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `vm_account_id` and `vm_project_id` labels to results of alerting and recording rules if `-clusterMode` is enabled. This improves [multitenant support in vmalert](https://docs.victoriametrics.com/vmalert/#multitenancy). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): minimize the time needed for reading large responses from scrape targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). This should reduce scrape durations for such targets as [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) running in a big Kubernetes cluster. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [sort_by_label_numeric](https://docs.victoriametrics.com/metricsql/#sort_by_label_numeric) and [sort_by_label_numeric_desc](https://docs.victoriametrics.com/metricsql/#sort_by_label_numeric_desc) functions for [numeric sort](https://www.gnu.org/software/coreutils/manual/html_node/Version-sort-is-not-the-same-as-numeric-sort.html) of input time series by the specified labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2938). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/): retry GCS operations for up to 3 minutes on temporary failures. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): add support for saving / restoring backups to / from [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029). +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): expose `vm_backup_in_flight` metric, which can be used for determining which backup types - latest, hourly, daily, weekly or monthly - are currently executed. +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add ability to extract JWT authorization token from non-standard HTTP header by passing it via `-auth.httpHeader` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3054). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `__meta_ec2_region` label for [ec2_sd_config](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs) in the same way as [Prometheus 2.39 does](https://github.com/prometheus/prometheus/pull/11326). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): accept data ingestion requests via paths starting from `/prometheus` prefix in the same way as [VictoriaMetrics does](https://docs.victoriametrics.com/#how-to-import-time-series-data). For example, `vmagent` now accepts Prometheus `remote_write` data via both `/api/v1/write` and `/prometheus/api/v1/write`. This simplifies switching between single-node VictoriaMetrics and `vmagent`. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `external_labels` from `global` section at `-promscrape.config` after the [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) is applied to scraped metrics. This aligns with Prometheus behaviour. Previously the `external_labels` were added to scrape targets, so they could be modified during relabeling. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying per-`-remoteWrite.url` limits for on-disk size for pending data via `-remoteWrite.maxDiskUsagePerURL` command-line flag. Thanks to @rbizos for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): log clear error when multiple identical `-storageNode` command-line flags are passed to `vmselect` or to `vminsert`. Previously these components were crashed with cryptic panic `metric ... is already registered` in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3076). -* BUGFIX: do not export stale metrics via [/federate api](./README.md#federation) after the staleness markers. Previously such metrics were exported with `NaN` values. this could break some setups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185). -* BUGFIX: export infinity numbers as `"Infinity"` strings at [/api/v1/export](./README.md#how-to-export-data-in-json-line-format), so they can be parsed by standard JSON parsers. Previously infinity numbers were exported as `Inf` values, which couldn't be parsed by standard JSON parsers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161). -* BUGFIX: [vmauth](./vmauth.md): properly handle request paths ending with `/` such as `/vmui/`. Previously `vmui` was dropping the trailing `/`, which could prevent from using `vmui` via `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752). -* BUGFIX: [vmagent](./vmagent.md): properly encode query params for aws signed requests, use `%20` instead of `+` as api requires. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171). -* BUGFIX: [vmagent](./vmagent.md): properly parse relabel config when regex ending with escaped `$`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d`. Previously the `sum_over_time(m[d])` could be improperly divided by smaller than `d` time range. See [rate_over_sum() docs](./MetricsQL.md#rate_over_sum) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate `increase(m[d])` over slow-changing counters with values smaller than 100. Previously [increase](./MetricsQL.md#increase) could return unexpectedly big results in this case. See [the related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3163). -* BUGFIX: [MetricsQL](./MetricsQL.md): ignore empty series when applying [limit_offset](./MetricsQL.md#limit_offset). It should improve queries with additional filters by value in expressions like `limit_offset(1,1, foo > 1)`. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [quantiles_over_time](./MetricsQL.md#quantiles_over_time) when the lookbehind window contains only a single sample. Previously an empty result was incorrectly returned in this case. -* BUGFIX: [vmui](./README.md#vmui): fix `RangeError: Maximum call stack size exceeded` error when the query returns too many data points at `Table` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3092/files). -* BUGFIX: [vmui](./README.md#vmui): fix workaround for adding more queries via URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169). -* BUGFIX: [vmalert](./vmalert.md): re-evaluate annotations per each alert evaluation. Previously, annotations were evaluated only on alert's value change. This could result in stale annotations in some cases described in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119). -* BUGFIX: prevent from excessive CPU usage when the storage enters [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). The previous fix in [v1.81.0](./CHANGELOG.md#v1810) wasn't complete. -* BUGFIX: [vmalert](./vmalert.md): change default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. Param `step` is added by vmalert to every rule evaluation request sent to datasource. Before this change, `step` was equal to group's evaluation interval by default. Param `step` for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. -* BUGFIX: properly calculate `vm_rows_scanned_per_query` histogram exported at `/metrics` page of `vmselect` and single-node VictoriaMetrics. Previously it could return misleadingly high numbers for [rollup functions](./MetricsQL.md#rollup-functions), which scan only a few samples on the provided lookbehind window in square brackets. For example, `increase(m[1d])` always scans only 2 rows (aka `raw samples`) per each returned time series. +* BUGFIX: do not export stale metrics via [/federate api](https://docs.victoriametrics.com/#federation) after the staleness markers. Previously such metrics were exported with `NaN` values. this could break some setups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185). +* BUGFIX: export infinity numbers as `"Infinity"` strings at [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format), so they can be parsed by standard JSON parsers. Previously infinity numbers were exported as `Inf` values, which couldn't be parsed by standard JSON parsers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly handle request paths ending with `/` such as `/vmui/`. Previously `vmui` was dropping the trailing `/`, which could prevent from using `vmui` via `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly encode query params for aws signed requests, use `%20` instead of `+` as api requires. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly parse relabel config when regex ending with escaped `$`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d`. Previously the `sum_over_time(m[d])` could be improperly divided by smaller than `d` time range. See [rate_over_sum() docs](https://docs.victoriametrics.com/metricsql/#rate_over_sum) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate `increase(m[d])` over slow-changing counters with values smaller than 100. Previously [increase](https://docs.victoriametrics.com/metricsql/#increase) could return unexpectedly big results in this case. See [the related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3163). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): ignore empty series when applying [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset). It should improve queries with additional filters by value in expressions like `limit_offset(1,1, foo > 1)`. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [quantiles_over_time](https://docs.victoriametrics.com/metricsql/#quantiles_over_time) when the lookbehind window contains only a single sample. Previously an empty result was incorrectly returned in this case. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `RangeError: Maximum call stack size exceeded` error when the query returns too many data points at `Table` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3092/files). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix workaround for adding more queries via URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): re-evaluate annotations per each alert evaluation. Previously, annotations were evaluated only on alert's value change. This could result in stale annotations in some cases described in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119). +* BUGFIX: prevent from excessive CPU usage when the storage enters [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). The previous fix in [v1.81.0](https://docs.victoriametrics.com/changelog/#v1810) wasn't complete. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): change default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. Param `step` is added by vmalert to every rule evaluation request sent to datasource. Before this change, `step` was equal to group's evaluation interval by default. Param `step` for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. +* BUGFIX: properly calculate `vm_rows_scanned_per_query` histogram exported at `/metrics` page of `vmselect` and single-node VictoriaMetrics. Previously it could return misleadingly high numbers for [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions), which scan only a few samples on the provided lookbehind window in square brackets. For example, `increase(m[1d])` always scans only 2 rows (aka `raw samples`) per each returned time series. ## [v1.81.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.2) Released at 2022-09-08 -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly calculate query results at `vmselect`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067). The issue has been introduced in [v1.81.0](./CHANGELOG.md#v1810). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly calculate query results at `vmselect`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067). The issue has been introduced in [v1.81.0](https://docs.victoriametrics.com/changelog/#v1810). ## [v1.81.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.1) Released at 2022-09-02 -**It isn't recommended to use VictoriaMetrics cluster v1.81.1 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067), which may result in incorrect query results. Upgrade to [v1.81.2](./CHANGELOG.md#v1812) instead.** +**It isn't recommended to use VictoriaMetrics cluster v1.81.1 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067), which may result in incorrect query results. Upgrade to [v1.81.2](https://docs.victoriametrics.com/changelog/#v1812) instead.** -* FEATURE: [MetricsQL](./MetricsQL.md): evaluate `q1`, ..., `qN` in parallel when calculating `union(q1, .., qN)`. Previously [union](./MetricsQL.md#union) args were evaluated sequentially. This could result in lower than expected performance. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): evaluate `q1`, ..., `qN` in parallel when calculating `union(q1, .., qN)`. Previously [union](https://docs.victoriametrics.com/metricsql/#union) args were evaluated sequentially. This could result in lower than expected performance. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fix potential panic at `vmselect` under high load, which has been introduced in [v1.81.0](./CHANGELOG.md#v1810). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fix potential panic at `vmselect` under high load, which has been introduced in [v1.81.0](https://docs.victoriametrics.com/changelog/#v1810). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058). ## [v1.81.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.81.0) -**It isn't recommended to use VictoriaMetrics cluster v1.81.0 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058), which may result in `vmselect` crashes under high load. Upgrade to [v1.81.2](./CHANGELOG.md#v1812) instead.** +**It isn't recommended to use VictoriaMetrics cluster v1.81.0 because of [the bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058), which may result in `vmselect` crashes under high load. Upgrade to [v1.81.2](https://docs.victoriametrics.com/changelog/#v1812) instead.** Released at 2022-08-31 -**Update note 1:** [vmalert](./vmalert.md) by default hides values of `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` in logs and at `http://vmalert:8880/flags` for security reasons. See the corresponding SECURITY change in the Changelog below for additional info. +**Update note 1:** [vmalert](https://docs.victoriametrics.com/vmalert/) by default hides values of `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` in logs and at `http://vmalert:8880/flags` for security reasons. See the corresponding SECURITY change in the Changelog below for additional info. -**Update note 2:** [vmalert](./vmalert.md) by default points alert source url to `/vmalert/alert?...` aka [web UI](./vmalert.md#web) instead of `/vmalert/api/v1/alert?...` aka JSON handler. The old behavior can be achieved by setting `-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}` command-line flag. +**Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert/) by default points alert source url to `/vmalert/alert?...` aka [web UI](https://docs.victoriametrics.com/vmalert/#web) instead of `/vmalert/api/v1/alert?...` aka JSON handler. The old behavior can be achieved by setting `-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}` command-line flag. -* SECURITY: [vmalert](./vmalert.md): do not expose `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flag values in logs and at `http://vmalert:8880/flags` page by default, since they may contain sensitive data such as auth keys. This aligns `vmalert` behaviour with [vmagent](./vmagent.md), which doesn't expose `-remoteWrite.url` command-line flag value in logs and at `http://vmagent:8429/flags` page by default. Specify `-remoteWrite.showURL`, `-remoteRead.showURL` and `-datasource.showURL` command-line flags for showing values for the corresponding `-*.url` flags in logs. Thanks to @mble for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2965). +* SECURITY: [vmalert](https://docs.victoriametrics.com/vmalert/): do not expose `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flag values in logs and at `http://vmalert:8880/flags` page by default, since they may contain sensitive data such as auth keys. This aligns `vmalert` behaviour with [vmagent](https://docs.victoriametrics.com/vmagent/), which doesn't expose `-remoteWrite.url` command-line flag value in logs and at `http://vmagent:8429/flags` page by default. Specify `-remoteWrite.showURL`, `-remoteRead.showURL` and `-datasource.showURL` command-line flags for showing values for the corresponding `-*.url` flags in logs. Thanks to @mble for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2965). * SECURITY: upgrade base docker image (alpine) from 3.16.1 to 3.16.2. See [alpine 3.16.2 release notes](https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html). -* FEATURE: return shorter error messages to Grafana and to other clients requesting [/api/v1/query](./keyConcepts.md#instant-query) and [/api/v1/query_range](./keyConcepts.md#range-query) endpoints. This should simplify reading these errors by humans. The long error message with full context is still written to logs. -* FEATURE: add the ability to fine-tune the number of points, which can be generated per each matching time series during [subquery](./MetricsQL.md#subqueries) evaluation. This can be done with the `-search.maxPointsSubqueryPerTimeseries` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922). -* FEATURE: [vmagent](./vmagent.md): improve the performance for relabeling rules with commonly used regular expressions in `regex` and `if` fields such as `some_string`, `prefix.*`, `prefix.+`, `foo|bar|baz`, `.*foo.*` and `.+foo.+`. -* FEATURE: [vmagent](./vmagent.md): reduce CPU usage when discovering big number of [Kubernetes targets](./sd_configs.md#kubernetes_sd_configs) with big number of labels and annotations. -* FEATURE: [vmagent](./vmagent.md): add ability to accept [multitenant](./Cluster-VictoriaMetrics.md#multitenancy) data via OpenTSDB `/api/put` protocol at `/insert//opentsdb/api/put` http endpoint if [multitenant support](./vmagent.md#multitenancy) is enabled at `vmagent`. Thanks to @chengjianyun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3015). -* FEATURE: [monitoring](./README.md#monitoring): expose `vm_hourly_series_limit_max_series`, `vm_hourly_series_limit_current_series`, `vm_daily_series_limit_max_series` and `vm_daily_series_limit_current_series` metrics when `-search.maxHourlySeries` or `-search.maxDailySeries` limits are set. This allows alerting when the number of unique series reaches the configured limits. See [these docs](./README.md#cardinality-limiter) for details. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): reduce the amounts of logging at `vmstorage` when `vmselect` connects/disconnects to `vmstorage`. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): improve performance for heavy queries on systems with many CPU cores. -* FEATURE: [vmagent](./vmagent.md): add ability to use `{{label_name}}` placeholders in the `replacement` option of relabeling rules. This simplifies constructing label values from multiple existing label values. See [these docs](./vmagent.md#relabeling-enhancements) for details. -* FEATURE: [vmagent](./vmagent.md): generate additional per-target metrics - `scrape_series_limit`, `scrape_series_current` and `scrape_series_limit_samples_dropped` if series limit is set according to [these docs](./vmagent.md#cardinality-limiter). This simplifies alerting on targets with the exceeded series limit. See [these docs](./vmagent.md#automatically-generated-metrics) for details on these metrics. -* FEATURE: [vmagent](./vmagent.md): add support for MX record types in [dns_sd_configs](./sd_configs.md#dns_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/10099). -* FEATURE: [vmagent](./vmagent.md): add `__meta_kubernetes_service_port_number` meta-label for `role: service` in [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/11002). -* FEATURE: [vmagent](./vmagent.md): add `__meta_kubernetes_pod_container_image` meta-label for `role: pod` in [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/11034). -* FEATURE: [vmagent](./vmagent.md): retry HTTP requests after some wait time during service discovery and during target scrapes if the server returns 429 HTTP status code (aka `Too many requests`). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). -* FEATURE: [vmui](./README.md#vmui): add a legend in the top right corner for shortcut keys. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2813). -* FEATURE: [vmalert](./vmalert.md): add `toTime()` template function in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/10993). See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/#numbers). -* FEATURE: [vmalert](./vmalert.md): add `$alertID` and `$groupID` template variables. These variables may be used for templating annotations or `-external.alert.source` command-line flag. See the full list of supported variables [here](./vmalert.md#templating). -* FEATURE: [vmalert](./vmalert.md): add `$activeAt` template variable. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2999). See the full list of supported variables [here](./vmalert.md#templating). Thanks to @laixintao for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3000). -* FEATURE: [vmalert](./vmalert.md): point alert source to [vmalert's UI](./vmalert.md#web) at `/vmalert/alert?...` instead of JSON handler at `/vmalert/api/v1/alert?...`. This improves user experience. The old behavior can be achieved by setting `-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}` command-line flag. +* FEATURE: return shorter error messages to Grafana and to other clients requesting [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) endpoints. This should simplify reading these errors by humans. The long error message with full context is still written to logs. +* FEATURE: add the ability to fine-tune the number of points, which can be generated per each matching time series during [subquery](https://docs.victoriametrics.com/metricsql/#subqueries) evaluation. This can be done with the `-search.maxPointsSubqueryPerTimeseries` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): improve the performance for relabeling rules with commonly used regular expressions in `regex` and `if` fields such as `some_string`, `prefix.*`, `prefix.+`, `foo|bar|baz`, `.*foo.*` and `.+foo.+`. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU usage when discovering big number of [Kubernetes targets](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) with big number of labels and annotations. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to accept [multitenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) data via OpenTSDB `/api/put` protocol at `/insert//opentsdb/api/put` http endpoint if [multitenant support](https://docs.victoriametrics.com/vmagent/#multitenancy) is enabled at `vmagent`. Thanks to @chengjianyun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3015). +* FEATURE: [monitoring](https://docs.victoriametrics.com/#monitoring): expose `vm_hourly_series_limit_max_series`, `vm_hourly_series_limit_current_series`, `vm_daily_series_limit_max_series` and `vm_daily_series_limit_current_series` metrics when `-search.maxHourlySeries` or `-search.maxDailySeries` limits are set. This allows alerting when the number of unique series reaches the configured limits. See [these docs](https://docs.victoriametrics.com/#cardinality-limiter) for details. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): reduce the amounts of logging at `vmstorage` when `vmselect` connects/disconnects to `vmstorage`. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): improve performance for heavy queries on systems with many CPU cores. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to use `{{label_name}}` placeholders in the `replacement` option of relabeling rules. This simplifies constructing label values from multiple existing label values. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): generate additional per-target metrics - `scrape_series_limit`, `scrape_series_current` and `scrape_series_limit_samples_dropped` if series limit is set according to [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). This simplifies alerting on targets with the exceeded series limit. See [these docs](https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics) for details on these metrics. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for MX record types in [dns_sd_configs](https://docs.victoriametrics.com/sd_configs/#dns_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/10099). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_service_port_number` meta-label for `role: service` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/11002). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_pod_container_image` meta-label for `role: pod` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/11034). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): retry HTTP requests after some wait time during service discovery and during target scrapes if the server returns 429 HTTP status code (aka `Too many requests`). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add a legend in the top right corner for shortcut keys. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2813). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `toTime()` template function in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/10993). See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/#numbers). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `$alertID` and `$groupID` template variables. These variables may be used for templating annotations or `-external.alert.source` command-line flag. See the full list of supported variables [here](https://docs.victoriametrics.com/vmalert/#templating). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `$activeAt` template variable. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2999). See the full list of supported variables [here](https://docs.victoriametrics.com/vmalert/#templating). Thanks to @laixintao for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3000). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): point alert source to [vmalert's UI](https://docs.victoriametrics.com/vmalert/#web) at `/vmalert/alert?...` instead of JSON handler at `/vmalert/api/v1/alert?...`. This improves user experience. The old behavior can be achieved by setting `-external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}` command-line flag. -* BUGFIX: prevent from excess CPU usage when the storage enters [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). -* BUGFIX: improve performance for requests to [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) when the filter in the `match[]` query arg matches small number of time series. The performance for this case has been reduced in [v1.78.0](./CHANGELOG.md#v1780). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533) issues. +* BUGFIX: prevent from excess CPU usage when the storage enters [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). +* BUGFIX: improve performance for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) when the filter in the `match[]` query arg matches small number of time series. The performance for this case has been reduced in [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533) issues. * BUGFIX: increase the default limit on the number of concurrent merges for small parts from 8 to 16. This should help resolving potential issues with heavy data ingestion. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673#issuecomment-1218185978) from @lukepalmer . -* BUGFIX: [MetricsQL](./MetricsQL.md): fix panic when incorrect arg is passed as `phi` into [histogram_quantiles](./MetricsQL.md#histogram_quantiles) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): fix panic when incorrect arg is passed as `phi` into [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026). ## [v1.80.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.80.0) Released at 2022-08-08 -* FEATURE: [vmalert](./vmalert.md): allow configuring additional HTTP request headers for `-datasource.url`, `-remoteWrite.url` and `-remoteRead.url` via `-datasource.headers`, `-remoteWrite.headers` and `-remoteRead.headers` command-line flags. Additional HTTP request headers also can be set on group level via `headers` param - see [these docs](./vmalert.md#groups) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860). -* FEATURE: [MetricsQL](./MetricsQL.md): execute left and right sides of certain operations in parallel. For example, `q1 or q2`, `aggr_func(q1) q2`, `q1 aggr_func(q1)`. This may improve query performance if VictoriaMetrics has enough free resources for parallel processing of both sides of the operation. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886). -* FEATURE: [vmauth](./vmagent.md): allow multiple sections with duplicate `username` but with different `password` values at `-auth.config` file. -* FEATURE: add ability to push internal metrics (e.g. metrics exposed at `/metrics` page) to the configured remote storage from all the VictoriaMetrics components. See [these docs](./README.md#push-metrics). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow configuring additional HTTP request headers for `-datasource.url`, `-remoteWrite.url` and `-remoteRead.url` via `-datasource.headers`, `-remoteWrite.headers` and `-remoteRead.headers` command-line flags. Additional HTTP request headers also can be set on group level via `headers` param - see [these docs](https://docs.victoriametrics.com/vmalert/#groups) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): execute left and right sides of certain operations in parallel. For example, `q1 or q2`, `aggr_func(q1) q2`, `q1 aggr_func(q1)`. This may improve query performance if VictoriaMetrics has enough free resources for parallel processing of both sides of the operation. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmagent/): allow multiple sections with duplicate `username` but with different `password` values at `-auth.config` file. +* FEATURE: add ability to push internal metrics (e.g. metrics exposed at `/metrics` page) to the configured remote storage from all the VictoriaMetrics components. See [these docs](https://docs.victoriametrics.com/#push-metrics). * FEATURE: improve performance for heavy queries over big number of time series on systems with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896). Thanks to @zqyzyq for [the idea](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b596ac3745314fcc170a14e3ded062971cf7ced2). * FEATURE: improve performance for registering new time series in `indexdb` by up to 50%. Thanks to @ahfuzhang for [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249). -* FEATURE: [vmagent](./vmagent.md): add ability to specify tenantID in target labels. In this case metrics from the given target are routed to the given `__tenant_id__`. See [these docs](./vmagent.md#multitenancy) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2943). -* FEATURE: [vmagent](./vmagent.md): add service discovery for [Yandex Cloud](https://cloud.yandex.com/en/). See [these docs](./sd_configs.md#yandexcloud_sd_configs) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386). -* FEATURE: [vmui](./README.md#vmui). Zoom in the graph by selecting the needed time range in the same way Grafana does. Hold `ctrl` (or `cmd` on MacOS) in order to move the graph to the left/right. Hold `ctrl` (or `cmd` on MacOS) and scroll up/down in order to zoom in/out the area under the cursor. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2812). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to specify tenantID in target labels. In this case metrics from the given target are routed to the given `__tenant_id__`. See [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2943). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add service discovery for [Yandex Cloud](https://cloud.yandex.com/en/). See [these docs](https://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui). Zoom in the graph by selecting the needed time range in the same way Grafana does. Hold `ctrl` (or `cmd` on MacOS) in order to move the graph to the left/right. Hold `ctrl` (or `cmd` on MacOS) and scroll up/down in order to zoom in/out the area under the cursor. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2812). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fix potential panic in [multi-level cluster setup](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) when top-level `vmselect` is configured with `-replicationFactor` bigger than 1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961). -* BUGFIX: [vmagent](./vmagent.md): properly handle custom `endpoint` value in [ec2_sd_configs](./sd_configs.md#ec2_sd_configs). It was ignored since [v1.77.0](./CHANGELOG.md#v1770) because of a bug in the implementation of [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2917). -* BUGFIX: [vmagent](./vmagent.md): add missing `__meta_kubernetes_ingress_class_name` meta-label for `role: ingress` service discovery in Kubernetes. See [this commit from Prometheus](https://github.com/prometheus/prometheus/commit/7e65ad3e432bd2837c17e3e63e85dcbcc30f4a8a). -* BUGFIX: [vmagent](./vmagent.md): allow stale responses from Consul service discovery (aka [consul_sd_configs](./sd_configs.md#consul_sd_configs)) by default in the same way as Prometheus does. This should reduce load on Consul when discovering big number of targets. Stale responses can be disabled by specifying `allow_stale: false` option in `consul_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): properly set `__meta_dockerswarm_container_label_*` labels instead of `__meta_dockerswarm_task_label_*` labels as Prometheus does. See [this issue](https://github.com/prometheus/prometheus/issues/9187). -* BUGFIX: [vmagent](./vmagent.md): set `up` metric to `0` for partial scrapes in [stream parsing mode](./vmagent.md#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus. -* BUGFIX: [vmagent](./vmagent.md): restart all the scrape jobs during [config reload](./vmagent.md#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884). -* BUGFIX: [vmagent](./vmagent.md): properly assume role with AWS ECS credentials. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2875). Thanks to @transacid for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2876). -* BUGFIX: [vmagent](./vmagent.md): do not split regex in [relabeling rules](./vmagent.md#relabeling) into multiple lines if it contains groups. This fixes [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928). -* BUGFIX: [MetricsQL](./MetricsQL.md): return series from `q1` if `q2` doesn't return matching time series in the query `q1 ifnot q2`. Previously series from `q1` weren't returned in this case. -* BUGFIX: [vmui](./README.md#vmui): properly show date picker at `Table` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fix potential panic in [multi-level cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) when top-level `vmselect` is configured with `-replicationFactor` bigger than 1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly handle custom `endpoint` value in [ec2_sd_configs](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs). It was ignored since [v1.77.0](https://docs.victoriametrics.com/changelog/#v1770) because of a bug in the implementation of [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2917). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): add missing `__meta_kubernetes_ingress_class_name` meta-label for `role: ingress` service discovery in Kubernetes. See [this commit from Prometheus](https://github.com/prometheus/prometheus/commit/7e65ad3e432bd2837c17e3e63e85dcbcc30f4a8a). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): allow stale responses from Consul service discovery (aka [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs)) by default in the same way as Prometheus does. This should reduce load on Consul when discovering big number of targets. Stale responses can be disabled by specifying `allow_stale: false` option in `consul_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): properly set `__meta_dockerswarm_container_label_*` labels instead of `__meta_dockerswarm_task_label_*` labels as Prometheus does. See [this issue](https://github.com/prometheus/prometheus/issues/9187). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): set `up` metric to `0` for partial scrapes in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): restart all the scrape jobs during [config reload](https://docs.victoriametrics.com/vmagent/#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly assume role with AWS ECS credentials. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2875). Thanks to @transacid for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2876). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not split regex in [relabeling rules](https://docs.victoriametrics.com/vmagent/#relabeling) into multiple lines if it contains groups. This fixes [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): return series from `q1` if `q2` doesn't return matching time series in the query `q1 ifnot q2`. Previously series from `q1` weren't returned in this case. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly show date picker at `Table` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874). * BUGFIX: properly generate http redirects if `-http.pathPrefix` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918). ## [v1.79.14](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.14) Released at 2023-08-12 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade Go builder from Go1.20.4 to Go1.21.0. * SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See [alpine 3.18.3 release notes](https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html). -* BUGFIX: [vmagent](./vmagent.md): properly apply `if` filters during [relabeling](./vmagent.md#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). -* BUGFIX: [vmalert](./vmalert.md): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). -* BUGFIX: [vmalert](./vmalert.md): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply `if` filters during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). ## [v1.79.13](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.13) Released at 2023-05-18 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See [the list of issues addressed in Go1.20.4](https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See [alpine 3.18.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html). @@ -416,13 +416,13 @@ The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHAN Released at 2023-04-06 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See [alpine 3.17.3 release notes](https://alpinelinux.org/posts/Alpine-3.17.3-released.html). * SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See [the list of issues addressed in Go1.20.3](https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): fix CPU and memory usage spikes when files pointed by [file_sd_config](./sd_configs.md#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix CPU and memory usage spikes when files pointed by [file_sd_config](https://docs.victoriametrics.com/sd_configs/#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). * BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023). * BUGFIX: verify response code when fetching configuration files via HTTP. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034). @@ -430,156 +430,156 @@ The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHAN Released at 2023-03-12 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See [the list of issues addressed in Go1.20.2](https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved). * BUGFIX: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3946). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](./CHANGELOG.md#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). * BUGFIX: prevent from possible `SIGBUS` crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for [the initial attempt to fix it](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927). ## [v1.79.10](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.10) Released at 2023-02-27 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -* BUGFIX: prevent from high CPU usage on the first UTC hour of the data. The issue has been introduced in [v1.79.5](./CHANGELOG.md#v1795) when fixing [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3309). +* BUGFIX: prevent from high CPU usage on the first UTC hour of the data. The issue has been introduced in [v1.79.5](https://docs.victoriametrics.com/changelog/#v1795) when fixing [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3309). ## [v1.79.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.9) Released at 2023-02-24 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See [alpine 3.17.2 release notes](https://alpinelinux.org/posts/Alpine-3.17.2-released.html). * SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See [the list of issues addressed in Go1.20.1](https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved). -* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). +* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). ## [v1.79.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.8) Released at 2023-02-03 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * BUGFIX: fix a bug, which could prevent background merges for the previous partitions until restart if the storage didn't have enough disk space for final deduplication and down-sampling. -* BUGFIX: [vmagent](./vmagent.md): update API version for [ec2_sd_configs](./sd_configs.md#ec2_sd_configs) to fix [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700) with missing `__meta_ec2_availability_zone_id` attribute. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fix panic on top-level vmselect nodes of [multi-level setup](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) when the `-replicationFactor` flag is set and request contains `trace` query parameter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3734). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). -* BUGFIX: [vmagent](./vmagent.md): suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. -* BUGFIX: [vmagent](./vmagent.md): consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly parse `M` and `Mi` suffixes as `1e6` multipliers in `1M` and `1Mi` numeric constants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664). The issue has been introduced in [v1.79.7](./CHANGELOG.md#v1797). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): update API version for [ec2_sd_configs](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs) to fix [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700) with missing `__meta_ec2_availability_zone_id` attribute. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fix panic on top-level vmselect nodes of [multi-level setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) when the `-replicationFactor` flag is set and request contains `trace` query parameter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3734). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly parse `M` and `Mi` suffixes as `1e6` multipliers in `1M` and `1Mi` numeric constants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664). The issue has been introduced in [v1.79.7](https://docs.victoriametrics.com/changelog/#v1797). ## [v1.79.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.7) Released at 2023-01-10 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -* BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during [data import](./README.md#how-to-import-time-series-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): properly encode `filters` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579) +* BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during [data import](https://docs.victoriametrics.com/#how-to-import-time-series-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): properly encode `filters` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579) * BUGFIX: allow specifying values bigger than 2GiB to the following command-line flag values on 32-bit architectures (`386` and `arm`): `-storage.minFreeDiskSpaceBytes` and `-remoteWrite.maxDiskUsagePerURL`. Previously values bigger than 2GiB were incorrectly truncated on these architectures. -* BUGFIX: [VictoriaMetrics enterprise](./enterprise.md): expose proper values for `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which were added at [v1.78.0](./CHANGELOG.md#v1780). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). -* BUGFIX: [DataDog protocol parser](./README.md#how-to-send-data-from-datadog-agent): do not re-use `host` and `device` fields from the previously parsed messages if these fields are missing in the currently parsed message. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). +* BUGFIX: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): expose proper values for `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which were added at [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). +* BUGFIX: [DataDog protocol parser](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent): do not re-use `host` and `device` fields from the previously parsed messages if these fields are missing in the currently parsed message. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432). ## [v1.79.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.6) Released at 2022-12-11 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: update Go builder from v1.19.3 to v1.19.4. See [the changelog](https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved). * SECURITY: update base Docker image for VictoriaMetrics components from Alpine 3.16.2 to Alpine v3.17.0. See [the changelog](https://alpinelinux.org/releases/). -* BUGFIX: [vmagent](./vmagent.md): fix the `The request did not have a subscription or a valid tenant level resource provider` error when discovering Azure targets with [azure_sd_configs](./sd_configs.md#azure_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247). -* BUGFIX: [vmagent](./vmagent.md): properly discover GCE zones when `filter` option is set at [gce_sd_configs](./sd_configs.md#gce_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202). -* BUGFIX: [vmalert](./vmalert.md): properly specify rule evaluation step during the [replay mode](./vmalert.md#rules-backfilling). The `step` value was previously overriden by `-datasource.queryStep` command-line flag. -* BUGFIX: [vmalert](./vmalert.md): properly return the error message from remote-write failures. Before, error was ignored and only `vmalert_remotewrite_errors_total` was incremented. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly return an empty result from [limit_offset](./MetricsQL.md#limit_offset) if the `offset` arg exceeds the number of inner time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix the `The request did not have a subscription or a valid tenant level resource provider` error when discovering Azure targets with [azure_sd_configs](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover GCE zones when `filter` option is set at [gce_sd_configs](https://docs.victoriametrics.com/sd_configs/#gce_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly specify rule evaluation step during the [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling). The `step` value was previously overriden by `-datasource.queryStep` command-line flag. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return the error message from remote-write failures. Before, error was ignored and only `vmalert_remotewrite_errors_total` was incremented. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly return an empty result from [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) if the `offset` arg exceeds the number of inner time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312). ## [v1.79.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.5) Released at 2022-11-10 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -**Update note 1:** [vmalert](./vmalert.md): the `crlfEscape` [template function](./vmalert.md#template-functions) becomes obsolete starting from this release. It can be safely removed from alerting templates, since `\n` chars are properly escaped with other `*Escape` functions now. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue for details. +**Update note 1:** [vmalert](https://docs.victoriametrics.com/vmalert/): the `crlfEscape` [template function](https://docs.victoriametrics.com/vmalert/#template-functions) becomes obsolete starting from this release. It can be safely removed from alerting templates, since `\n` chars are properly escaped with other `*Escape` functions now. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue for details. * SECURITY: update Go builder to v1.19.3. This fixes [CVE-2022 security issue](https://github.com/golang/go/issues/56328). See [the changelog](https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved). * BUGFIX: properly register new time series in per-day inverted index if they were ingested during the last 10 seconds of the day. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3309). Thanks to @lmarszal for the bugreport and for the [initial fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3320). -* BUGFIX: properly accept [OpenTSDB telnet put lines](./README.md#sending-data-via-telnet-put-protocol) without tags without the need to specify the trailing whitespace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](./MetricsQL.md#histogram_quantile) and [histogram_share](./MetricsQL.md#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225). -* BUGFIX: [vmalert](./vmalert.md): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level. -* BUGFIX: [vmalert](./vmalert.md): properly escape string passed to `quotesEscape` [template function](./vmalert.md#template-functions), so it can be safely embedded into JSON string. This makes obsolete the `crlfEscape` function. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue. +* BUGFIX: properly accept [OpenTSDB telnet put lines](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol) without tags without the need to specify the trailing whitespace. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) and [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly escape string passed to `quotesEscape` [template function](https://docs.victoriametrics.com/vmalert/#template-functions), so it can be safely embedded into JSON string. This makes obsolete the `crlfEscape` function. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890) issue. * BUGFIX: `vmselect`: expose missing metric `vm_cache_size_max_bytes{type="promql/rollupResult"}` . This metric is used for monitoring rollup cache usage with the query `vm_cache_size_bytes{type="promql/rollupResult"} / vm_cache_size_max_bytes{type="promql/rollupResult"}` in the same way as this is done for other cache types. ## [v1.79.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.4) Released at 2022-10-07 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -**Update note 1:** [vmalert](./vmalert.md) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. +**Update note 1:** [vmalert](https://docs.victoriametrics.com/vmalert/) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. * FEATURE: expose `vmagent_remotewrite_queues` metric and use it in alerting rules in order to improve the detection of remote storage connection saturation. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2871). -* BUGFIX: do not export stale metrics via [/federate api](./README.md#federation) after the staleness markers. Previously such metrics were exported with `NaN` values. this could break some setups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185). -* BUGFIX: [vmauth](./vmauth.md): properly handle request paths ending with `/` such as `/vmui/`. Previously `vmui` was dropping the trailing `/`, which could prevent from using `vmui` via `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752). -* BUGFIX: [vmagent](./vmagent.md): properly encode query params for aws signed requests, use `%20` instead of `+` as api requires. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d`. Previously the `sum_over_time(m[d])` could be improperly divided by smaller than `d` time range. See [rate_over_sum() docs](./MetricsQL.md#rate_over_sum) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate `increase(m[d])` over slow-changing counters with values smaller than 100. Previously [increase](./MetricsQL.md#increase) could return unexpectedly big results in this case. See [the related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3163). -* BUGFIX: [MetricsQL](./MetricsQL.md): ignore empty series when applying [limit_offset](./MetricsQL.md#limit_offset). It should improve queries with additional filters by value in expressions like `limit_offset(1,1, foo > 1)`. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [quantiles_over_time](./MetricsQL.md#quantiles_over_time) when the lookbehind window contains only a single sample. Previously an empty result was incorrectly returned in this case. -* BUGFIX: [vmui](./README.md#vmui): fix `RangeError: Maximum call stack size exceeded` error when the query returns too many data points at `Table` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3092/files). -* BUGFIX: [vmalert](./vmalert.md): re-evaluate annotations per each alert evaluation. Previously, annotations were evaluated only on alert's value change. This could result in stale annotations in some cases described in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119). -* BUGFIX: prevent from excessive CPU usage when the storage enters [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). The previous fix in [v1.79.3](./CHANGELOG.md#v1793) wasn't complete. -* BUGFIX: [vmalert](./vmalert.md): change default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. Param `step` is added by vmalert to every rule evaluation request sent to datasource. Before this change, `step` was equal to group's evaluation interval by default. Param `step` for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. -* BUGFIX: properly calculate `vm_rows_scanned_per_query` histogram exported at `/metrics` page of `vmselect` and single-node VictoriaMetrics. Previously it could return misleadingly high numbers for [rollup functions](./MetricsQL.md#rollup-functions), which scan only a few samples on the provided lookbehind window in square brackets. For example, `increase(m[1d])` always scans only 2 rows (aka `raw samples`) per each returned time series. +* BUGFIX: do not export stale metrics via [/federate api](https://docs.victoriametrics.com/#federation) after the staleness markers. Previously such metrics were exported with `NaN` values. this could break some setups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly handle request paths ending with `/` such as `/vmui/`. Previously `vmui` was dropping the trailing `/`, which could prevent from using `vmui` via `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly encode query params for aws signed requests, use `%20` instead of `+` as api requires. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d`. Previously the `sum_over_time(m[d])` could be improperly divided by smaller than `d` time range. See [rate_over_sum() docs](https://docs.victoriametrics.com/metricsql/#rate_over_sum) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate `increase(m[d])` over slow-changing counters with values smaller than 100. Previously [increase](https://docs.victoriametrics.com/metricsql/#increase) could return unexpectedly big results in this case. See [the related issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3163). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): ignore empty series when applying [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset). It should improve queries with additional filters by value in expressions like `limit_offset(1,1, foo > 1)`. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [quantiles_over_time](https://docs.victoriametrics.com/metricsql/#quantiles_over_time) when the lookbehind window contains only a single sample. Previously an empty result was incorrectly returned in this case. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `RangeError: Maximum call stack size exceeded` error when the query returns too many data points at `Table` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3092/files). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): re-evaluate annotations per each alert evaluation. Previously, annotations were evaluated only on alert's value change. This could result in stale annotations in some cases described in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119). +* BUGFIX: prevent from excessive CPU usage when the storage enters [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). The previous fix in [v1.79.3](https://docs.victoriametrics.com/changelog/#v1793) wasn't complete. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): change default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. Param `step` is added by vmalert to every rule evaluation request sent to datasource. Before this change, `step` was equal to group's evaluation interval by default. Param `step` for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. +* BUGFIX: properly calculate `vm_rows_scanned_per_query` histogram exported at `/metrics` page of `vmselect` and single-node VictoriaMetrics. Previously it could return misleadingly high numbers for [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions), which scan only a few samples on the provided lookbehind window in square brackets. For example, `increase(m[1d])` always scans only 2 rows (aka `raw samples`) per each returned time series. ## [v1.79.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.3) Released at 2022-08-30 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -* SECURITY: [vmalert](./vmalert.md): do not expose `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flag values in logs and at `http://vmalert:8880/flags` page by default, since they may contain sensitive data such as auth keys. This aligns `vmalert` behaviour with [vmagent](./vmagent.md), which doesn't expose `-remoteWrite.url` command-line flag value in logs and at `http://vmagent:8429/flags` page by default. Specify `-remoteWrite.showURL`, `-remoteRead.showURL` and `-datasource.showURL` command-line flags for showing values for the corresponding `-*.url` flags in logs. Thanks to @mble for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2965). +* SECURITY: [vmalert](https://docs.victoriametrics.com/vmalert/): do not expose `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flag values in logs and at `http://vmalert:8880/flags` page by default, since they may contain sensitive data such as auth keys. This aligns `vmalert` behaviour with [vmagent](https://docs.victoriametrics.com/vmagent/), which doesn't expose `-remoteWrite.url` command-line flag value in logs and at `http://vmagent:8429/flags` page by default. Specify `-remoteWrite.showURL`, `-remoteRead.showURL` and `-datasource.showURL` command-line flags for showing values for the corresponding `-*.url` flags in logs. Thanks to @mble for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2965). * SECURITY: upgrade base docker image (alpine) from 3.16.1 to 3.16.2. See [alpine 3.16.2 release notes](https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html). -* BUGFIX: prevent from excess CPU usage when the storage enters [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). -* BUGFIX: improve performance for requests to [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) when the filter in the `match[]` query arg matches small number of time series. The performance for this case has been reduced in [v1.78.0](./CHANGELOG.md#v1780). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533) issues. +* BUGFIX: prevent from excess CPU usage when the storage enters [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). +* BUGFIX: improve performance for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) when the filter in the `match[]` query arg matches small number of time series. The performance for this case has been reduced in [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533) issues. * BUGFIX: increase the default limit on the number of concurrent merges for small parts from 8 to 16. This should help resolving potential issues with heavy data ingestion. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673#issuecomment-1218185978) from @lukepalmer . -* BUGFIX: [MetricsQL](./MetricsQL.md): fix panic when incorrect arg is passed as `phi` into [histogram_quantiles](./MetricsQL.md#histogram_quantiles) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): fix panic when incorrect arg is passed as `phi` into [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026). ## [v1.79.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.2) Released at 2022-08-08 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fix potential panic in [multi-level cluster setup](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) when top-level `vmselect` is configured with `-replicationFactor` bigger than 1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961). -* BUGFIX: [vmagent](./vmagent.md): properly handle custom `endpoint` value in [ec2_sd_configs](./sd_configs.md#ec2_sd_configs). It was ignored since [v1.77.0](./CHANGELOG.md#v1770) because of a bug in the implementation of [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). -* BUGFIX: [vmagent](./vmagent.md): add missing `__meta_kubernetes_ingress_class_name` meta-label for `role: ingress` service discovery in Kubernetes. See [this commit from Prometheus](https://github.com/prometheus/prometheus/commit/7e65ad3e432bd2837c17e3e63e85dcbcc30f4a8a). -* BUGFIX: [vmagent](./vmagent.md): allow stale responses from Consul service discovery (aka [consul_sd_configs](./sd_configs.md#consul_sd_configs)) by default in the same way as Prometheus does. This should reduce load on Consul when discovering big number of targets. Stale responses can be disabled by specifying `allow_stale: false` option in `consul_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): properly set `__meta_dockerswarm_container_label_*` labels instead of `__meta_dockerswarm_task_label_*` labels as Prometheus does. See [this issue](https://github.com/prometheus/prometheus/issues/9187). -* BUGFIX: [vmagent](./vmagent.md): set `up` metric to `0` for partial scrapes in [stream parsing mode](./vmagent.md#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus. -* BUGFIX: [vmagent](./vmagent.md): restart all the scrape jobs during [config reload](./vmagent.md#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884). -* BUGFIX: [vmagent](./vmagent.md): properly assume role with AWS ECS credentials. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2875). Thanks to @transacid for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2876). -* BUGFIX: [vmagent](./vmagent.md): do not split regex in [relabeling rules](./vmagent.md#relabeling) into multiple lines if it contains groups. This fixes [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928). -* BUGFIX: [MetricsQL](./MetricsQL.md): return series from `q1` if `q2` doesn't return matching time series in the query `q1 ifnot q2`. Previously series from `q1` weren't returned in this case. -* BUGFIX: [vmui](./README.md#vmui): properly show date picker at `Table` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fix potential panic in [multi-level cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) when top-level `vmselect` is configured with `-replicationFactor` bigger than 1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly handle custom `endpoint` value in [ec2_sd_configs](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs). It was ignored since [v1.77.0](https://docs.victoriametrics.com/changelog/#v1770) because of a bug in the implementation of [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): add missing `__meta_kubernetes_ingress_class_name` meta-label for `role: ingress` service discovery in Kubernetes. See [this commit from Prometheus](https://github.com/prometheus/prometheus/commit/7e65ad3e432bd2837c17e3e63e85dcbcc30f4a8a). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): allow stale responses from Consul service discovery (aka [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs)) by default in the same way as Prometheus does. This should reduce load on Consul when discovering big number of targets. Stale responses can be disabled by specifying `allow_stale: false` option in `consul_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): properly set `__meta_dockerswarm_container_label_*` labels instead of `__meta_dockerswarm_task_label_*` labels as Prometheus does. See [this issue](https://github.com/prometheus/prometheus/issues/9187). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): set `up` metric to `0` for partial scrapes in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): restart all the scrape jobs during [config reload](https://docs.victoriametrics.com/vmagent/#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly assume role with AWS ECS credentials. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2875). Thanks to @transacid for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2876). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not split regex in [relabeling rules](https://docs.victoriametrics.com/vmagent/#relabeling) into multiple lines if it contains groups. This fixes [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): return series from `q1` if `q2` doesn't return matching time series in the query `q1 ifnot q2`. Previously series from `q1` weren't returned in this case. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly show date picker at `Table` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874). * BUGFIX: properly generate http redirects if `-http.pathPrefix` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918). @@ -587,8 +587,8 @@ The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHAN Released at 2022-08-02 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** * SECURITY: upgrade base docker image (alpine) from 3.16.0 to 3.16.1 . See [alpine 3.16.1 release notes](https://alpinelinux.org/posts/Alpine-3.16.1-released.html). @@ -597,28 +597,28 @@ The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHAN Released at 2022-07-14 -**v1.79.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.79.x line will be supported for at least 12 months since [v1.79.0](./CHANGELOG.md#v1790) release** +**v1.79.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/changelog/#v1790) release** **Update note 1:** this release introduces backwards-incompatible changes to `vm_partial_results_total` metric by changing its labels to be consistent with `vm_requests_total` metric. If you use alerting rules or Grafana dashboards, which rely on this metric, then they must be updated. The official dashboards for VictoriaMetrics don't use this metric. -**Update note 2:** [vmalert](./vmalert.md) adds `/vmalert/` prefix to [web urls](./vmalert.md#web) according to [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825). This may affect `vmalert` instances with non-empty `-http.pathPrefix` command-line flag. After the update, configuring this flag is no longer needed. Here's [why](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005). +**Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert/) adds `/vmalert/` prefix to [web urls](https://docs.victoriametrics.com/vmalert/#web) according to [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825). This may affect `vmalert` instances with non-empty `-http.pathPrefix` command-line flag. After the update, configuring this flag is no longer needed. Here's [why](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005). -**Update note 3:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics because of added ability to query `vmselect` data from other `vmselect` nodes - see [these docs](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup), so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases at any time. +**Update note 3:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics because of added ability to query `vmselect` data from other `vmselect` nodes - see [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup), so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases at any time. -**Update note 4:** this release removes support of deprecated in [1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0) param `extra_filter_labels` from [vmalert's](./vmalert.md) groups definition. This deprecated param was replaced with [params](./vmalert.md#url-params). +**Update note 4:** this release removes support of deprecated in [1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0) param `extra_filter_labels` from [vmalert's](https://docs.victoriametrics.com/vmalert/) groups definition. This deprecated param was replaced with [params](https://docs.victoriametrics.com/vmalert/#url-params). **Update note 5:** this release changes naming for published linux binaries at [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Now names for binaries for all the supported platforms match the following template - `$(APP_NAME)-$(GOOS)-$(GOARCH)-$(VERSION).tar.gz`. For example, `victoria-metrics-linux-amd64-v1.79.0.tar.gz`. Previously linux binaries didn't have `$(GOOS)` part, e.g. they had the name `victoria-metrics-amd64-v1.79.0.tar.gz`. Please update automation scripts for upgrading VictoriaMetrics releases according to this change. -* FEATURE: [vmagent](./vmagent.md): add [azure_sd_configs](./sd_configs.md#azure_sd_configs) service discovery mechanism. It allows discovering Virtual Machines at [Azure Cloud](https://azure.microsoft.com/en-us/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364). -* FEATURE: [vmalert](./vmalert.md): deprecate alert's status link `/api/v1///status` in favour of `api/v1/alert?group_id=&alert_id="`. The old alert's status link is still supported, but will be removed in future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825). -* FEATURE: [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md): add support for querying lower-level `vmselect` nodes from upper-level `vmselect` nodes. This makes possible to build multi-level cluster setups for global querying view and HA purposes without the need to use [Promxy](https://github.com/jacksontj/promxy). See [these docs](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2778). -* FEATURE: add `-search.setLookbackToStep` command-line flag, which enables InfluxDB-like gap filling during querying. See [these docs](./guides/migrate-from-influx.md) for details. -* FEATURE: [vmui](./README.md#vmui): add an UI for [query tracing](./README.md#query-tracing). It can be enabled by clicking `trace query` checkbox and re-running the query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2703). -* FEATURE: [vmagent](./vmagent.md): add `-remoteWrite.headers` command-line option for specifying optional HTTP headers to send to the configured `-remoteWrite.url`. For example, `-remoteWrite.headers='Foo:Bar^^Baz:x'` would send `Foo: Bar` and `Baz: x` HTTP headers with every request to `-remoteWrite.url`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805). -* FEATURE: [vmagent](./vmagent.md): push per-target `scrape_samples_limit` metric to the configured `-remoteWrite.url` if `sample_limit` option is set for this target in [scrape_configs](./sd_configs.md#scrape_configs). See [this feature request](https://github.com/VictoriaMetrics/operator/issues/497). -* FEATURE: [vmagent](./vmagent.md): attach node-level labels to [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs) targets if `attach_metadata: {"node": true}` is set for `role: endpoints` and `role: endpointslice`. This is a feature backport from Prometheus 2.37 - see [this pull request](https://github.com/prometheus/prometheus/pull/10759). -* FEATURE: [vmagent](./vmagent.md): add ability to specify additional HTTP headers to send to scrape targets via `headers` section in `scrape_configs`. This can be used when the scrape target requires custom authorization and authentication like in [this stackoverflow question](https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header). For example, the following config instructs sending `My-Auth: top-secret` and `TenantID: FooBar` headers with each request to `http://host123:8080/metrics`: +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add [azure_sd_configs](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs) service discovery mechanism. It allows discovering Virtual Machines at [Azure Cloud](https://azure.microsoft.com/en-us/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): deprecate alert's status link `/api/v1///status` in favour of `api/v1/alert?group_id=&alert_id="`. The old alert's status link is still supported, but will be removed in future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825). +* FEATURE: [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/): add support for querying lower-level `vmselect` nodes from upper-level `vmselect` nodes. This makes possible to build multi-level cluster setups for global querying view and HA purposes without the need to use [Promxy](https://github.com/jacksontj/promxy). See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2778). +* FEATURE: add `-search.setLookbackToStep` command-line flag, which enables InfluxDB-like gap filling during querying. See [these docs](https://docs.victoriametrics.com/guides/migrate-from-influx.html) for details. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add an UI for [query tracing](https://docs.victoriametrics.com/#query-tracing). It can be enabled by clicking `trace query` checkbox and re-running the query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2703). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-remoteWrite.headers` command-line option for specifying optional HTTP headers to send to the configured `-remoteWrite.url`. For example, `-remoteWrite.headers='Foo:Bar^^Baz:x'` would send `Foo: Bar` and `Baz: x` HTTP headers with every request to `-remoteWrite.url`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): push per-target `scrape_samples_limit` metric to the configured `-remoteWrite.url` if `sample_limit` option is set for this target in [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs). See [this feature request](https://github.com/VictoriaMetrics/operator/issues/497). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): attach node-level labels to [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) targets if `attach_metadata: {"node": true}` is set for `role: endpoints` and `role: endpointslice`. This is a feature backport from Prometheus 2.37 - see [this pull request](https://github.com/prometheus/prometheus/pull/10759). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to specify additional HTTP headers to send to scrape targets via `headers` section in `scrape_configs`. This can be used when the scrape target requires custom authorization and authentication like in [this stackoverflow question](https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header). For example, the following config instructs sending `My-Auth: top-secret` and `TenantID: FooBar` headers with each request to `http://host123:8080/metrics`: ```yaml scrape_configs: @@ -630,49 +630,49 @@ scrape_configs: - targets: ["host123:8080"] ``` -* FEATURE: add ability to pass `limit` query arg to `api/v1/series` endpoint. This can be used if only a sample of up to `limit` series must be returned from the endpoint. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841) and [these docs](./README.md#prometheus-querying-api-enhancements). -* FEATURE: [query tracing](./Single-Server-VictoriaMetrics.md#query-tracing): show timestamps in query traces in human-readable format (aka `RFC3339` in UTC timezone) instead of milliseconds since Unix epoch. For example, `2022-06-27T10:32:54.506Z` instead of `1656325974506`. This improves traces' readability. -* FEATURE: improve performance of [/api/v1/series](./url-examples.md#apiv1series) requests, which return big number of time series. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): improve query performance when [replication is enabled](./Cluster-VictoriaMetrics.md#replication-and-data-safety). -* FEATURE: [MetricsQL](./MetricsQL.md): properly handle partial counter resets in [remove_resets](./MetricsQL.md#remove_resets) function. Now `remove_resets(sum(m))` should returns the expected increasing line when some time series matching `m` disappear on the selected time range. Previously such a query would return horizontal line after the disappeared series. +* FEATURE: add ability to pass `limit` query arg to `api/v1/series` endpoint. This can be used if only a sample of up to `limit` series must be returned from the endpoint. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841) and [these docs](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements). +* FEATURE: [query tracing](https://docs.victoriametrics.com/single-server-victoriametrics/#query-tracing): show timestamps in query traces in human-readable format (aka `RFC3339` in UTC timezone) instead of milliseconds since Unix epoch. For example, `2022-06-27T10:32:54.506Z` instead of `1656325974506`. This improves traces' readability. +* FEATURE: improve performance of [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) requests, which return big number of time series. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): improve query performance when [replication is enabled](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle partial counter resets in [remove_resets](https://docs.victoriametrics.com/metricsql/#remove_resets) function. Now `remove_resets(sum(m))` should returns the expected increasing line when some time series matching `m` disappear on the selected time range. Previously such a query would return horizontal line after the disappeared series. * FEATURE: expose `vm_next_retention_seconds` metric at `http://victoriametrics:8428/metrics`, which shows the number of seconds left until the next `indexdb` rotation. Thanks to @guidao for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2863). * FEATURE: expose additional histogram metrics at `http://victoriametrics:8428/metrics`, which may help understanding query workload: * `vm_rows_read_per_query` - the number of raw samples read per query. - * `vm_rows_scanned_per_query` - the number of raw samples scanned per query. This number can exceed `vm_rows_read_per_query` if `step` query arg passed to [/api/v1/query_range](./keyConcepts.md#range-query) is smaller than the lookbehind window set in square brackets of [rollup function](./MetricsQL.md#rollup-functions). For example, if `increase(some_metric[1h])` is executed with the `step=5m`, then the same raw samples on a hour time range are scanned `1h/5m=12` times. See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986) for details. + * `vm_rows_scanned_per_query` - the number of raw samples scanned per query. This number can exceed `vm_rows_read_per_query` if `step` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) is smaller than the lookbehind window set in square brackets of [rollup function](https://docs.victoriametrics.com/metricsql/#rollup-functions). For example, if `increase(some_metric[1h])` is executed with the `step=5m`, then the same raw samples on a hour time range are scanned `1h/5m=12` times. See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986) for details. * `vm_rows_read_per_series` - the number of raw samples read per queried series. * `vm_series_read_per_query` - the number of series read per query. * FEATURE: publish binaries for FreeBSD and OpenBSD at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). -* FEATURE: [vmui](./README.md#vmui): allow selecting the needed columns at table view. This functionally may help when the selected time series contain many different labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2817) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2867). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): allow selecting the needed columns at table view. This functionally may help when the selected time series contain many different labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2817) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2867). * BUGFIX: consistently name binaries at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) in the form `$(APP_NAME)-$(GOOS)-$(GOARCH)-$(VERSION).tar.gz`. For example, `victoria-metrics-linux-amd64-v1.79.0.tar.gz`. Previously the `$(GOOS)` part was missing in binaries for Linux. -* BUGFIX: [vmalert](./vmalert.md): allow using `__name__` label (aka [metric name](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)) in alerting annotations. For example: +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): allow using `__name__` label (aka [metric name](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)) in alerting annotations. For example: ```sh {{ $labels.__name__ }}: Too high connection number for "{{ $labels.instance }} ``` -* BUGFIX: limit max memory occupied by the cache, which stores parsed regular expressions. Previously too long regular expressions passed in [MetricsQL queries](./MetricsQL.md) could result in big amounts of used memory (e.g. multiple of gigabytes). Now the max cache size for parsed regexps is limited to a a few megabytes. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle partial counter resets when calculating [rate](./MetricsQL.md#rate), [irate](./MetricsQL.md#irate) and [increase](./MetricsQL.md#increase) functions. Previously these functions could return zero values after partial counter resets until the counter increases to the last value before partial counter reset. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate [histogram_quantile](./MetricsQL.md#histogram_quantile) over Prometheus buckets with unexpected values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly evaluate [timezone_offset](./MetricsQL.md#timezone_offset) function over time range covering time zone offset switches. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771). -* BUGFIX: [vmagent](./vmagent.md): properly add service-level labels (`__meta_kubernetes_service_*`) to discovered targets for `role: endpointslice` in [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs). Previously these labels were missing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823). -* BUGFIX: [vmagent](./vmagent.md): make sure that [stale markers](./vmagent.md#prometheus-staleness-markers) are generated with the actual timestamp when unsuccessful scrape occurs. This should prevent from possible time series overlap on scrape target restart in dynamic environments such as Kubernetes. -* BUGFIX: [vmagent](./vmagent.md): properly reload changed `-promscrape.config` file when `-promscrape.configCheckInterval` option is set. The changed config file wasn't reloaded in this case since [v1.69.0](./CHANGELOG_2021.md#v1690). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2786). Thanks to @ttyv for the fix. -* BUGFIX: [vmagent](./vmagent.md): properly set `Host` header during target scraping when `proxy_url` is set to http proxy. Previously the `Host` header was set to the proxy hostname instead of the target hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): assume that the response is complete if `-search.denyPartialResponse` is enabled and up to `-replicationFactor - 1` `vmstorage` nodes are unavailable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767). -* BUGFIX: [vmselect](./README.md#vmselect): update `vm_partial_results_total` metric labels to be consistent with `vm_requests_total` labels. -* BUGFIX: accept tags without values when reading data in [DataDog format](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent). Thanks to @PerGon for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839). -* BUGFIX: [vmui](./README.md#vmui): properly pass the end of the selected time range to `time` query arg to [/api/v1/query](./keyConcepts.md#instant-query) when displaying the requested data in JSON and table views. Previously the `time` query arg wasn't set, so `/api/v1/query` was always returning query results for the current time regardless of the selected time range. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2781). -* BUGFIX: [vmui](./README.md#vmui): allow clicking on the suggestion from autocomplete list. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2804). -* BUGFIX: [vmui](./README.md#vmui): apply the selected time range in date picker only after clicking the `Apply` button. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2811). +* BUGFIX: limit max memory occupied by the cache, which stores parsed regular expressions. Previously too long regular expressions passed in [MetricsQL queries](https://docs.victoriametrics.com/metricsql/) could result in big amounts of used memory (e.g. multiple of gigabytes). Now the max cache size for parsed regexps is limited to a a few megabytes. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle partial counter resets when calculating [rate](https://docs.victoriametrics.com/metricsql/#rate), [irate](https://docs.victoriametrics.com/metricsql/#irate) and [increase](https://docs.victoriametrics.com/metricsql/#increase) functions. Previously these functions could return zero values after partial counter resets until the counter increases to the last value before partial counter reset. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) over Prometheus buckets with unexpected values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly evaluate [timezone_offset](https://docs.victoriametrics.com/metricsql/#timezone_offset) function over time range covering time zone offset switches. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly add service-level labels (`__meta_kubernetes_service_*`) to discovered targets for `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously these labels were missing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): make sure that [stale markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) are generated with the actual timestamp when unsuccessful scrape occurs. This should prevent from possible time series overlap on scrape target restart in dynamic environments such as Kubernetes. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly reload changed `-promscrape.config` file when `-promscrape.configCheckInterval` option is set. The changed config file wasn't reloaded in this case since [v1.69.0](https://docs.victoriametrics.com/changelog_2021/#v1690). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2786). Thanks to @ttyv for the fix. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `Host` header during target scraping when `proxy_url` is set to http proxy. Previously the `Host` header was set to the proxy hostname instead of the target hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): assume that the response is complete if `-search.denyPartialResponse` is enabled and up to `-replicationFactor - 1` `vmstorage` nodes are unavailable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767). +* BUGFIX: [vmselect](https://docs.victoriametrics.com/#vmselect): update `vm_partial_results_total` metric labels to be consistent with `vm_requests_total` labels. +* BUGFIX: accept tags without values when reading data in [DataDog format](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-datadog-agent). Thanks to @PerGon for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly pass the end of the selected time range to `time` query arg to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) when displaying the requested data in JSON and table views. Previously the `time` query arg wasn't set, so `/api/v1/query` was always returning query results for the current time regardless of the selected time range. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2781). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): allow clicking on the suggestion from autocomplete list. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2804). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): apply the selected time range in date picker only after clicking the `Apply` button. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2811). ## [v1.78.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.1) Released at 2022-07-08 -**Update notes:** it is recommended [clearing caches](./README.md#cache-removal) after the upgrade from [v1.78.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.0) in order to immediately fix the issue for newly ingested data. Otherwise the issue may exist for newly ingested data for up to a day after the upgrade. +**Update notes:** it is recommended [clearing caches](https://docs.victoriametrics.com/#cache-removal) after the upgrade from [v1.78.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.0) in order to immediately fix the issue for newly ingested data. Otherwise the issue may exist for newly ingested data for up to a day after the upgrade. * BUGFIX: properly register time series in per-day inverted index. Previously some series could miss registration in the per-day inverted index. This could result in missing time series during querying. The issue has been introduced in [v1.78.0](#v1780). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2798) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2793) issues. @@ -682,121 +682,121 @@ Released at 2022-06-20 **Warning (2022-07-03):** VictoriaMetrics v1.78.0 contains a bug, which may result in missing time series during queries. It is recommended upgrading to [v1.78.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.78.1), which fixes the bug. -**Update notes:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics because of added [query tracing](./Single-Server-VictoriaMetrics.md#query-tracing), so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases. +**Update notes:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics because of added [query tracing](https://docs.victoriametrics.com/single-server-victoriametrics/#query-tracing), so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases. * SECURITY: add `-flagsAuthKey` command-line flag for protecting `/flags` endpoint from unauthorized access. Though this endpoint already hides values for command-line flags with `key` and `password` substrings in their names, other sensitive information could be exposed there. See [This issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753). -* FEATURE: support query tracing, which allows determining bottlenecks during query processing. See [these docs](./Single-Server-VictoriaMetrics.md#query-tracing) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403). -* FEATURE: [vmui](./README.md#vmui): add `cardinality` tab, which can help identifying the source of [high cardinality](./FAQ.md#what-is-high-cardinality) and [high churn rate](./FAQ.md#what-is-high-churn-rate) issues. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2730) feature requests and [these docs](./README.md#cardinality-explorer). -* FEATURE: [vmui](./README.md#vmui): small UX enhancements according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2638). +* FEATURE: support query tracing, which allows determining bottlenecks during query processing. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#query-tracing) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `cardinality` tab, which can help identifying the source of [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) and [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2730) feature requests and [these docs](https://docs.victoriametrics.com/#cardinality-explorer). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): small UX enhancements according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2638). * FEATURE: allow overriding default limits for in-memory cache `indexdb/tagFilters` via flag `-storage.cacheSizeIndexDBTagFilters`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663). * FEATURE: add support of `lowercase` and `uppercase` relabeling actions in the same way as [Prometheus 2.36.0 does](https://github.com/prometheus/prometheus/releases/tag/v2.36.0). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664). * FEATURE: add ability to change the `indexdb` rotation timezone offset via `-retentionTimezoneOffset` command-line flag. Previously it was performed at 4am UTC time. This could lead to performance degradation in the middle of the day when VictoriaMetrics runs in time zones located too far from UTC. Thanks to @cnych for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574). * FEATURE: limit the number of background merge threads on systems with big number of CPU cores by default. This increases the max size of parts, which can be created during background merge when `-storageDataPath` directory has limited free disk space. This may improve on-disk data compression efficiency and query performance. The limits can be tuned if needed with `-smallMergeConcurrency` and `-bigMergeConcurrency` command-line flags. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673). -* FEATURE: accept optional `limit` query arg at [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) for limiting the number of sample entries returned from these endpoints. See [these docs](./README.md#prometheus-querying-api-enhancements). -* FEATURE: optimize performance for [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) endpoints when `match[]`, `extra_label` or `extra_filters[]` query args are passed to these endpoints. This should help with [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533). -* FEATURE: [vmalert](./vmalert.md): support `limit` param per-group for limiting number of produced samples per each rule. Thanks to @Howie59 for [implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2676). -* FEATURE: [vmalert](./vmalert.md): remove dependency on Internet access at [web API pages](./vmalert.md#web). Previously the functionality and the layout of these pages was broken without Internet access. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594). -* FEATURE: [vmalert](./vmalert.md): send alerts to the configured notifiers in parallel. Previously alerts were sent to notifiers sequentially. This could delay sending pending alerts when notifier blocks on the currently sent alert. -* FEATURE: [vmagent](./vmagent.md): implement the `http://vmagent:8429/service-discovery` page in the same way as Prometheus does. This page shows the original labels for all the discovered targets alongside the resulting labels after the relabeling. This simplifies service discovery debugging. -* FEATURE: [vmagent](./vmagent.md): remove dependency on Internet access at `http://vmagent:8429/targets` page. Previously the page layout was broken without Internet access. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594). -* FEATURE: [vmagent](./vmagent.md): add support for `kubeconfig_file` option at [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). It may be useful for Kubernetes monitoring by `vmagent` outside Kubernetes cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1464). -* FEATURE: [vmagent](./vmagent.md): expose `/api/v1/status/config` endpoint in the same way as Prometheus does. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/api/#config). -* FEATURE: [vmagent](./vmagent.md): add `-promscrape.suppressScrapeErrorsDelay` command-line flag, which can be used for delaying and aggregating the logging of per-target scrape errors. This may reduce the amounts of logs when `vmagent` scrapes many unreliable targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575). Thanks to @jelmd for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576). -* FEATURE: [vmagent](./vmagent.md): add `-promscrape.cluster.name` command-line flag, which allows proper data de-duplication when the same target is scraped from multiple [vmagent clusters](./vmagent.md#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679). -* FEATURE: [vmagent](./vmagent.md): add `action: graphite` relabeling rules optimized for extracting labels from Graphite-style metric names. See [these docs](./vmagent.md#graphite-relabeling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737). -* FEATURE: [VictoriaMetrics enterprise](./enterprise.md): expose `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which can be used for tracking the progress of initial [downsampling](./README.md#downsampling) for historical data. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): do not spend up to 5 seconds when trying to connect to unavailable `vmstorage` nodes. This should improve query latency when some of `vmstorage` nodes aren't available. Expose `vm_tcpdialer_addr_available{addr="..."}` metric at `http://vmselect:8481/metrics` for determining whether the given `addr` is available for establishing new connections. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711#issuecomment-1160363187). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `-vmstorageDialTimeout` command-line flags to `vmselect` and `vminsert` for tuning the maximum duration for connection establishing to `vmstorage` nodes. This should help resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711). +* FEATURE: accept optional `limit` query arg at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) for limiting the number of sample entries returned from these endpoints. See [these docs](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements). +* FEATURE: optimize performance for [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) endpoints when `match[]`, `extra_label` or `extra_filters[]` query args are passed to these endpoints. This should help with [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support `limit` param per-group for limiting number of produced samples per each rule. Thanks to @Howie59 for [implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2676). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): remove dependency on Internet access at [web API pages](https://docs.victoriametrics.com/vmalert/#web). Previously the functionality and the layout of these pages was broken without Internet access. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): send alerts to the configured notifiers in parallel. Previously alerts were sent to notifiers sequentially. This could delay sending pending alerts when notifier blocks on the currently sent alert. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): implement the `http://vmagent:8429/service-discovery` page in the same way as Prometheus does. This page shows the original labels for all the discovered targets alongside the resulting labels after the relabeling. This simplifies service discovery debugging. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): remove dependency on Internet access at `http://vmagent:8429/targets` page. Previously the page layout was broken without Internet access. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `kubeconfig_file` option at [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). It may be useful for Kubernetes monitoring by `vmagent` outside Kubernetes cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1464). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `/api/v1/status/config` endpoint in the same way as Prometheus does. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/api/#config). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-promscrape.suppressScrapeErrorsDelay` command-line flag, which can be used for delaying and aggregating the logging of per-target scrape errors. This may reduce the amounts of logs when `vmagent` scrapes many unreliable targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575). Thanks to @jelmd for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-promscrape.cluster.name` command-line flag, which allows proper data de-duplication when the same target is scraped from multiple [vmagent clusters](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `action: graphite` relabeling rules optimized for extracting labels from Graphite-style metric names. See [these docs](https://docs.victoriametrics.com/vmagent/#graphite-relabeling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737). +* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): expose `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which can be used for tracking the progress of initial [downsampling](https://docs.victoriametrics.com/#downsampling) for historical data. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): do not spend up to 5 seconds when trying to connect to unavailable `vmstorage` nodes. This should improve query latency when some of `vmstorage` nodes aren't available. Expose `vm_tcpdialer_addr_available{addr="..."}` metric at `http://vmselect:8481/metrics` for determining whether the given `addr` is available for establishing new connections. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711#issuecomment-1160363187). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `-vmstorageDialTimeout` command-line flags to `vmselect` and `vminsert` for tuning the maximum duration for connection establishing to `vmstorage` nodes. This should help resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711). -* BUGFIX: support for data ingestion in [DataDog format](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent) from legacy clients / agents. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670). Thanks to @elProxy for the fix. -* BUGFIX: [vmagent](./vmagent.md): do not expose `vm_promscrape_service_discovery_duration_seconds_bucket` metric for unused service discovery types. This reduces the number of metrics exported at `http://vmagent:8429/metrics`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671). -* BUGFIX: [vmalert](./vmalert.md): properly apply `alert_relabel_configs` relabeling rules to `-notifier.config` according to [these docs](./vmalert.md#notifier-configuration-file). Thanks to @spectvtor for [the bugfix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2633). -* BUGFIX: [vmalert](./vmalert.md): properly add `Content-Encoding: snappy`, `Content-Type: application/x-protobuf` and `X-Prometheus-Remote-Write-Version: 0.1.0` request headers when `vmalert` sends [evaluated recording rules' data](./vmalert.md#recording-rules) to `-remoteWrite.url`. These headers are needed by some remote storage systems in order to properly decode snappy-encoded request body. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2685) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2701) pull requests. Thanks to @manji-0 for th fix. +* BUGFIX: support for data ingestion in [DataDog format](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-datadog-agent) from legacy clients / agents. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670). Thanks to @elProxy for the fix. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not expose `vm_promscrape_service_discovery_duration_seconds_bucket` metric for unused service discovery types. This reduces the number of metrics exported at `http://vmagent:8429/metrics`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly apply `alert_relabel_configs` relabeling rules to `-notifier.config` according to [these docs](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file). Thanks to @spectvtor for [the bugfix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2633). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly add `Content-Encoding: snappy`, `Content-Type: application/x-protobuf` and `X-Prometheus-Remote-Write-Version: 0.1.0` request headers when `vmalert` sends [evaluated recording rules' data](https://docs.victoriametrics.com/vmalert/#recording-rules) to `-remoteWrite.url`. These headers are needed by some remote storage systems in order to properly decode snappy-encoded request body. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2685) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2701) pull requests. Thanks to @manji-0 for th fix. * BUGFIX: deny [background merge](https://valyala.medium.com/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) when the storage enters read-only mode, e.g. when free disk space becomes lower than `-storage.minFreeDiskSpaceBytes`. Background merge needs additional disk space, so it could result in `no space left on device` errors. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603). -* BUGFIX: [vmui](./README.md#vmui): properly apply the selected time range when auto-refresh is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2693). -* BUGFIX: [vmui](./README.md#vmui): properly update the url with vmui state when new query is entered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2692). -* BUGFIX: [Graphite render API](./README.md#graphite-render-api-usage): properly calculate sample timestamps when `moving*()` functions such as [movingAverage()](https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.movingAverage) are applied over [summarize()](https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.summarize). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly apply the selected time range when auto-refresh is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2693). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly update the url with vmui state when new query is entered. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2692). +* BUGFIX: [Graphite render API](https://docs.victoriametrics.com/#graphite-render-api-usage): properly calculate sample timestamps when `moving*()` functions such as [movingAverage()](https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.movingAverage) are applied over [summarize()](https://graphite.readthedocs.io/en/stable/functions.html#graphite.render.functions.summarize). * BUGFIX: limit the `end` query arg value to `+2 days` in the future at `/api/v1/*` endpoints, because VictoriaMetrics doesn't allow storing samples with timestamps bigger than +2 days in the future. This should help resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2669). * BUGFIX: properly register time series in per-day inverted index during the first hour after `indexdb` rotation. Previously this could lead to missing time series during querying if these time series stopped receiving new samples during the first hour after `indexdb` rotation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698). -* BUGFIX: do not register new series when `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits were reached. Previously samples for new series weren't added to the database when the [cardinality limit](./README.md#cardinality-limiter) was reached, but series were still registered in the inverted index (aka `indexdb`). This could lead to unbound `indexdb` growth during [high churn rate](./FAQ.md#what-is-high-churn-rate). +* BUGFIX: do not register new series when `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits were reached. Previously samples for new series weren't added to the database when the [cardinality limit](https://docs.victoriametrics.com/#cardinality-limiter) was reached, but series were still registered in the inverted index (aka `indexdb`). This could lead to unbound `indexdb` growth during [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). ## [v1.77.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.2) Released at 2022-05-21 -* FEATURE: [vmalert](./vmalert.md): support [reusable templates](https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#defining-reusable-templates) for rules annotations. The path to the template files can be specified via `-rule.templates` flag. See more about this feature [here](./vmalert.md#reusable-templates). Thanks to @AndrewChubatiuk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2532). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2510). -* FEATURE: [vmalert](./vmalert.md): expose `vmalert_iteration_interval_seconds` metric at `http://vmalert:8880/metrics`. This metric shows the configured per-group evaluation interval. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2618). -* FEATURE: [vmctl](./vmctl.md): add `influx-prometheus-mode` command-line flag, which allows to restore the original time series written from Prometheus into InfluxDB during data migration from InfluxDB to VictoriaMetrics. See [this feature request](https://github.com/VictoriaMetrics/vmctl/issues/8). Thanks to @mback2k for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2545). -* FEATURE: [vmagent](./vmagent.md): add ability to specify AWS service name when issuing requests to AWS api. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2605). Thanks to @transacid for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2604). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support [reusable templates](https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#defining-reusable-templates) for rules annotations. The path to the template files can be specified via `-rule.templates` flag. See more about this feature [here](https://docs.victoriametrics.com/vmalert/#reusable-templates). Thanks to @AndrewChubatiuk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2532). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2510). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): expose `vmalert_iteration_interval_seconds` metric at `http://vmalert:8880/metrics`. This metric shows the configured per-group evaluation interval. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2618). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `influx-prometheus-mode` command-line flag, which allows to restore the original time series written from Prometheus into InfluxDB during data migration from InfluxDB to VictoriaMetrics. See [this feature request](https://github.com/VictoriaMetrics/vmctl/issues/8). Thanks to @mback2k for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2545). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to specify AWS service name when issuing requests to AWS api. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2605). Thanks to @transacid for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2604). -* BUGFIX: [vmagent](./vmagent.md): fix a bug, which could lead to incomplete discovery of scrape targets in Kubernetes (aka `kubernetes_sd_config`). the bug has been introduced in [v1.77.0](./CHANGELOG.md#v1770). -* BUGFIX: [vmalert](./vmalert.md): support `scalar` result type in response. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2607). -* BUGFIX: [vmalert](./vmalert.md): support strings in `humanize.*` template function in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2569). -* BUGFIX: [vmalert](./vmalert.md): proxy `/rules` requests to vmalert from Grafana's alerting UI. This removes errors in Grafana's UI for Grafana versions older than `8.5.*`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2583) -* BUGFIX: [vmalert](./vmalert.md): do not add `/api/v1/query` suffix to `-datasource.url` if `-remoteRead.disablePathAppend` command-line flag is set. Previously this flag was applied only to `-remoteRead.url`, which could confuse users. -* BUGFIX: [vmalert](./vmalert.md): prevent from possible resource leak on config update, which could lead to the slowdown of `vmalert` over time. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2577). -* BUGFIX: [MetricsQL](./MetricsQL.md): do not return values from [label_value()](./MetricsQL.md#label_value) function if the original time series has no values at the selected timestamps. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): limit the number of concurrently established connections from vmselect to vmstorage. This should prevent from potentially high spikes in the number of established connections after temporary slowdown in connection handshake procedure between vmselect and vmstorage because of spikes in workload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552). -* BUGFIX: [vmctl](./vmctl.md): fix build for Solaris / SmartOS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322#issuecomment-1120276146). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix a bug, which could lead to incomplete discovery of scrape targets in Kubernetes (aka `kubernetes_sd_config`). the bug has been introduced in [v1.77.0](https://docs.victoriametrics.com/changelog/#v1770). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): support `scalar` result type in response. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2607). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): support strings in `humanize.*` template function in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2569). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): proxy `/rules` requests to vmalert from Grafana's alerting UI. This removes errors in Grafana's UI for Grafana versions older than `8.5.*`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2583) +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): do not add `/api/v1/query` suffix to `-datasource.url` if `-remoteRead.disablePathAppend` command-line flag is set. Previously this flag was applied only to `-remoteRead.url`, which could confuse users. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): prevent from possible resource leak on config update, which could lead to the slowdown of `vmalert` over time. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2577). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): do not return values from [label_value()](https://docs.victoriametrics.com/metricsql/#label_value) function if the original time series has no values at the selected timestamps. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): limit the number of concurrently established connections from vmselect to vmstorage. This should prevent from potentially high spikes in the number of established connections after temporary slowdown in connection handshake procedure between vmselect and vmstorage because of spikes in workload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix build for Solaris / SmartOS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322#issuecomment-1120276146). ## [v1.77.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.1) Released at 2022-05-07 -* FEATURE: [vmagent](./vmagent.md): add ability to specify filters for Availability Zones in [ec2_sd_config](./sd_configs.md#ec2_sd_configs) via `az_filters` section. This section can contain AZ-specific set of filters in the same way as the existing `filters` section, which is used for filtering EC2 instances. The list of supported AZ-specific filters is available [here](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html). -* FEATURE: [vmagent](./vmagent.md): expose `vmagent_remotewrite_global_rows_pushed_before_relabel_total` and `vmagent_remotewrite_rows_pushed_after_relabel_total` metrics at `http://vmagent:8429/metrics`, which can be used for monitoring the rate of rows (aka samples) pushed to remote storage before and after the relabeling via `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`. See [relabeling docs](./vmagent.md#relabeling) for details. -* FEATURE: [vmctl](./vmctl.md): add ability to skip `db` label during InfluxDB data import when `influx-skip-database-label` option is used. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2544). Thanks to @mback2k . +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to specify filters for Availability Zones in [ec2_sd_config](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs) via `az_filters` section. This section can contain AZ-specific set of filters in the same way as the existing `filters` section, which is used for filtering EC2 instances. The list of supported AZ-specific filters is available [here](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `vmagent_remotewrite_global_rows_pushed_before_relabel_total` and `vmagent_remotewrite_rows_pushed_after_relabel_total` metrics at `http://vmagent:8429/metrics`, which can be used for monitoring the rate of rows (aka samples) pushed to remote storage before and after the relabeling via `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`. See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) for details. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add ability to skip `db` label during InfluxDB data import when `influx-skip-database-label` option is used. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2544). Thanks to @mback2k . -* BUGFIX: [vmagent](./vmagent.md): properly process passwords and secrets specified in the file pointed by `-promscrape.config` command-line flag. All the passwords and secrets were mistakenly replaced with `` string in `v1.77.0`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2550). -* BUGFIX: [vmagent](./vmagent.md): rename `vmagent_remote_write_rate_limit_reached_total` metric to `vmagent_remotewrite_rate_limit_reached_total`, so its name is consistent with the rest of `vmagent_remotewrite_` metrics. -* BUGFIX: [vmagent](./vmagent.md): rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics. -* BUGFIX: [vmctl](./vmctl.md): properly import InfluxDB measurements if they contain `db` tag. Previously this could result in incomplete import of measurement tags. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2536). Thanks to @mback2k for the bugfix. -* BUGFIX: [vmui](./README.md#vmui): do not reset the selected relative time range when entering new query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402#issuecomment-1115817302). -* BUGFIX: [vmbackup](./vmbackup.md): disallow writing backups to `-storageDataPath` directory, since this directory is managed solely by VictoriaMetrics or `vmstorage`. Other apps shouldn't write into this directory. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2503). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly process passwords and secrets specified in the file pointed by `-promscrape.config` command-line flag. All the passwords and secrets were mistakenly replaced with `` string in `v1.77.0`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2550). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): rename `vmagent_remote_write_rate_limit_reached_total` metric to `vmagent_remotewrite_rate_limit_reached_total`, so its name is consistent with the rest of `vmagent_remotewrite_` metrics. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): properly import InfluxDB measurements if they contain `db` tag. Previously this could result in incomplete import of measurement tags. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2536). Thanks to @mback2k for the bugfix. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not reset the selected relative time range when entering new query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402#issuecomment-1115817302). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): disallow writing backups to `-storageDataPath` directory, since this directory is managed solely by VictoriaMetrics or `vmstorage`. Other apps shouldn't write into this directory. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2503). * BUGFIX: do not allow setting `-retentionPeriod` smaller than one day, since VictoriaMetrics doesn't support properly such small retention periods. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2496). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): do not drop samples routed to readonly `vmstorage` nodes if `-dropSamplesOnOverload` command-line flag is set. Try re-routing them to healthy `vmstorage` nodes instead. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2478). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): do not drop samples routed to readonly `vmstorage` nodes if `-dropSamplesOnOverload` command-line flag is set. Try re-routing them to healthy `vmstorage` nodes instead. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2478). ## [v1.77.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.0) Released at 2022-05-05 -* FEATURE: [vmagent](./vmagent.md): add support for sending data to remote storage with AWS sigv4 authorization. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). -* FEATURE: [vmagent](./vmagent.md): allow filtering targets by target url and by target labels with [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) on `http://vmagent:8429/targets` page. This may be useful when `vmagent` scrapes big number of targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1796). -* FEATURE: [vmagent](./vmagent.md): reduce `-promscrape.config` reload duration when the config contains big number of jobs (aka [scrape_configs](./sd_configs.md#scrape_configs) sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270). -* FEATURE: [vmagent](./vmagent.md): improve service discovery speed for big number of scrape targets. This should help when `vmagent` discovers big number of targets (e.g. thousands) in Kubernetes cluster. The service discovery speed now should scale with the number of CPU cores available to `vmagent`. -* FEATURE: [vmagent](./vmagent.md): add ability to attach node-level labels and annotations to discovered Kubernetes pod targets in the same way as Prometheus 2.35 does. See [this feature request](https://github.com/prometheus/prometheus/issues/9510) and [this pull request](https://github.com/prometheus/prometheus/pull/10080). -* FEATURE: [vmagent](./vmagent.md): add support for `tls_config` and `proxy_url` options at `oauth2` section in the same way as Prometheus does. See [oauth2 docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2). -* FEATURE: [vmagent](./vmagent.md): add support for `min_version` option at `tls_config` section in the same way as Prometheus does. See [tls_config docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config). -* FEATURE: [vmagent](./vmagent.md): expose `vmagent_remotewrite_rate_limit` metric at `http://vmagent:8429/metrics`, which can be used for alerting rules such as `rate(vmagent_remotewrite_conn_bytes_written_total) / vmagent_remotewrite_rate_limit > 0.8` when `-remoteWrite.rateLimit` command-line flag is set. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2521). -* FEATURE: [vmalert](./vmalert.md): add support for DNS-based discovery for notifiers in the same way as Prometheus does (aka `dns_sd_configs`). See [these docs](./vmalert.md#notifier-configuration-file) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2460). -* FEATURE: [vmalert](./vmalert.md): add `-replay.disableProgressBar` command-line flag, which allows disabling progressbar in [rules' backfilling mode](./vmalert.md#rules-backfilling). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1761). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for sending data to remote storage with AWS sigv4 authorization. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow filtering targets by target url and by target labels with [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) on `http://vmagent:8429/targets` page. This may be useful when `vmagent` scrapes big number of targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1796). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce `-promscrape.config` reload duration when the config contains big number of jobs (aka [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): improve service discovery speed for big number of scrape targets. This should help when `vmagent` discovers big number of targets (e.g. thousands) in Kubernetes cluster. The service discovery speed now should scale with the number of CPU cores available to `vmagent`. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to attach node-level labels and annotations to discovered Kubernetes pod targets in the same way as Prometheus 2.35 does. See [this feature request](https://github.com/prometheus/prometheus/issues/9510) and [this pull request](https://github.com/prometheus/prometheus/pull/10080). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `tls_config` and `proxy_url` options at `oauth2` section in the same way as Prometheus does. See [oauth2 docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for `min_version` option at `tls_config` section in the same way as Prometheus does. See [tls_config docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `vmagent_remotewrite_rate_limit` metric at `http://vmagent:8429/metrics`, which can be used for alerting rules such as `rate(vmagent_remotewrite_conn_bytes_written_total) / vmagent_remotewrite_rate_limit > 0.8` when `-remoteWrite.rateLimit` command-line flag is set. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2521). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add support for DNS-based discovery for notifiers in the same way as Prometheus does (aka `dns_sd_configs`). See [these docs](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2460). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-replay.disableProgressBar` command-line flag, which allows disabling progressbar in [rules' backfilling mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1761). * FEATURE: allow specifying TLS cipher suites for incoming https requests via `-tlsCipherSuites` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404). -* FEATURE: allow specifying TLS cipher suites for mTLS connections between cluster components via `-cluster.tlsCipherSuites` command-line flag. See [these docs](./Cluster-VictoriaMetrics.md#mtls-protection). +* FEATURE: allow specifying TLS cipher suites for mTLS connections between cluster components via `-cluster.tlsCipherSuites` command-line flag. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection). * FEATURE: vmstorage: add `-snapshotsMaxAge` command-line flag for automatic removal of snapshots older than the given age. -* FEATURE: [vmui](./README.md#vmui): show an empty graph on the selected time range when there is no data on it. Previously `No data to show` placeholder was shown instead of the graph in this case. This prevented from zooming and scrolling of such a graph. -* FEATURE: [vmui](./README.md#vmui): show the selected `last N minutes/hours/days` in the top right corner. Previously the `start - end` duration was shown instead, which could be hard to interpret. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402). -* FEATURE: [vmui](./README.md#vmui): execute the query when `enter` button is pressed in the same way as Prometheus does. Multi-line query can be entered by pressing `shift-enter` in the query input field. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show an empty graph on the selected time range when there is no data on it. Previously `No data to show` placeholder was shown instead of the graph in this case. This prevented from zooming and scrolling of such a graph. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show the selected `last N minutes/hours/days` in the top right corner. Previously the `start - end` duration was shown instead, which could be hard to interpret. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): execute the query when `enter` button is pressed in the same way as Prometheus does. Multi-line query can be entered by pressing `shift-enter` in the query input field. * FEATURE: expose `vm_indexdb_items_added_total` and `vm_indexdb_items_added_size_bytes_total` counters at `/metrics` page, which can be used for monitoring the rate for addition of new entries in `indexdb` (aka `inverted index`) alongside the total size in bytes for the added entries. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471). -* FEATURE: [vmctl](./vmctl.md): show data processing speed during data migration. -* FEATURE: [MetricsQL](./MetricsQL.md): add `drop_common_labels()` function, which drops common `label="name"` pairs from the passed time series. See [these docs](./MetricsQL.md#drop_common_labels). -* FEATURE: [MetricsQL](./MetricsQL.md): add `tlast_change_over_time(m[d])` function, which returns the timestamp of the last change of `m` on the given lookbehind window `d`. See [these docs](./MetricsQL.md#tlast_change_over_time). -* FEATURE: leave the last raw sample per each `-dedup.minScrapeInterval` discrete interval when the [deduplication](./README.md#deduplication) is enabled. This aligns better with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness) comparing to the previous behaviour when the first sample per each `-dedup.minScrapeInterval` was left. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add ability to disable peer TLS certificate verification with `-cluster.tlsInsecureSkipVerify` command-line flag. See [mTLS docs](./Cluster-VictoriaMetrics.md#mtls-protection) for details. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2490). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): show data processing speed during data migration. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `drop_common_labels()` function, which drops common `label="name"` pairs from the passed time series. See [these docs](https://docs.victoriametrics.com/metricsql/#drop_common_labels). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `tlast_change_over_time(m[d])` function, which returns the timestamp of the last change of `m` on the given lookbehind window `d`. See [these docs](https://docs.victoriametrics.com/metricsql/#tlast_change_over_time). +* FEATURE: leave the last raw sample per each `-dedup.minScrapeInterval` discrete interval when the [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. This aligns better with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness) comparing to the previous behaviour when the first sample per each `-dedup.minScrapeInterval` was left. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add ability to disable peer TLS certificate verification with `-cluster.tlsInsecureSkipVerify` command-line flag. See [mTLS docs](https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection) for details. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2490). * FEATURE: add a handler for `/api/v1/status/buildinfo` endpoint, which is used by Grafana starting from v8.5.0 . See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2515). * FEATURE: add ability to proxy alerting API requests from Grafana to vmalert by passing `-vmalert.proxyURL` command-line flag to single-node VictoriaMetrics or to `vmselect` at cluster version of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739). -* BUGFIX: export staleness markers as `null` values from [JSON export API](./README.md#how-to-export-data-in-json-line-format). Previously they were exported as `NaN` values. This could break the exported JSON parsing, since `NaN` values aren't supported by [JSON specification](https://www.json.org/). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): close `vmselect->vmstorage` connections if they were idle for more than 30 seconds. Expose `vm_tcpdialer_conns_idle` metric at `http://vmselect:8481/metrics` with the number of idle connections to `vmstorage`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508). -* BUGFIX: [vmctl](./vmctl.md): return non-zero exit code on error. This allows handling `vmctl` errors in shell scripts. Previously `vmctl` was returning 0 exit code on error. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2322). -* BUGFIX: [vmctl](./vmctl.md): prevent from indefinite hang on `Ctrl+C`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2491). -* BUGFIX: [vmagent](./vmagent.md): properly show `scrape_timeout` and `scrape_interval` options at `http://vmagent:8429/config` page. Previously these options weren't displayed even if they were set in `-promscrape.config`. -* BUGFIX: [vmagent](./vmagent.md): handle non-standard http redirect status codes, which may be returned by scrape targets, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482). -* BUGFIX: [vmalert](./vmalert.md): skip template execution during rules' validation. This should prevent from `error evaluating annotation template` errors when some template functions expect non-empty args. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2514). -* BUGFIX: [vmalert](./vmalert.md): fixed truncating alerts expression in table, updated table cell layout. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2484). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle joins on time series filtered by values. For example, `kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1)`. This query could result in `duplicate time series on the right side` error even if `==1` filter leaves only a single time series per `(namespace,pod)` labels. Now such query is properly executed. -* BUGFIX: [MetricsQL](./MetricsQL.md): properly handle `scalar default vector`, `scalar if vector` and `scalar ifnot vector` queries. Previously such queries could return unexpected results from the `vector` part. +* BUGFIX: export staleness markers as `null` values from [JSON export API](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format). Previously they were exported as `NaN` values. This could break the exported JSON parsing, since `NaN` values aren't supported by [JSON specification](https://www.json.org/). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): close `vmselect->vmstorage` connections if they were idle for more than 30 seconds. Expose `vm_tcpdialer_conns_idle` metric at `http://vmselect:8481/metrics` with the number of idle connections to `vmstorage`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): return non-zero exit code on error. This allows handling `vmctl` errors in shell scripts. Previously `vmctl` was returning 0 exit code on error. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2322). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): prevent from indefinite hang on `Ctrl+C`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2491). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly show `scrape_timeout` and `scrape_interval` options at `http://vmagent:8429/config` page. Previously these options weren't displayed even if they were set in `-promscrape.config`. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): handle non-standard http redirect status codes, which may be returned by scrape targets, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): skip template execution during rules' validation. This should prevent from `error evaluating annotation template` errors when some template functions expect non-empty args. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2514). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fixed truncating alerts expression in table, updated table cell layout. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2484). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle joins on time series filtered by values. For example, `kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1)`. This query could result in `duplicate time series on the right side` error even if `==1` filter leaves only a single time series per `(namespace,pod)` labels. Now such query is properly executed. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly handle `scalar default vector`, `scalar if vector` and `scalar ifnot vector` queries. Previously such queries could return unexpected results from the `vector` part. * BUGFIX: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): take into account `indexdb` when calculating disk space usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2368). @@ -806,14 +806,14 @@ Released at 2022-04-12 **Update notes:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics, so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases. -* FEATURE: [vmalert](./vmalert.md): add support for `alert_relabel_configs` option at `-notifier.config`. This option allows configuring relabeling rules for alerts before sending them to configured notifiers. See [these docs](./vmalert.md#notifier-configuration-file) for details. -* FEATURE: [vmagent](./vmalert.md): allow passing StatefulSet pod names to `-promscrape.cluster.memberNum` command-line flag. In this case the member number is automatically extracted from the pod name, which must end with the number in the range `0 ... promscrape.cluster.membersCount-1`. For example, `vmagent-0`, `vmagent-1`, etc. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359) and [these docs](./vmagent.md#scraping-big-number-of-targets). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add support for `alert_relabel_configs` option at `-notifier.config`. This option allows configuring relabeling rules for alerts before sending them to configured notifiers. See [these docs](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmalert/): allow passing StatefulSet pod names to `-promscrape.cluster.memberNum` command-line flag. In this case the member number is automatically extracted from the pod name, which must end with the number in the range `0 ... promscrape.cluster.membersCount-1`. For example, `vmagent-0`, `vmagent-1`, etc. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359) and [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly propagate limits at `-search.max*` command-line flags from `vminsert` to `vmstorage`. The limits are `-search.maxUniqueTimeseries`, `-search.maxSeries`, `-search.maxFederateSeries`, `-search.maxExportSeries`, `-search.maxGraphiteSeries` and `-search.maxTSDBStatusSeries`. They weren't propagated to `vmstorage` because of the bug. These limits were introduced in [v1.76.0](./CHANGELOG.md#v1760). See [this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2450). -* BUGFIX: fix goroutine leak and possible deadlock when importing invalid data via [native binary format](./README.md#how-to-import-data-in-native-format). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2423). -* BUGFIX: [Graphite Render API](./README.md#graphite-render-api-usage): properly calculate [hitCount](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.hitcount) function. Previously it could return empty results if there were no original samples in some parts of the selected time range. -* BUGFIX: [MetricsQL](./MetricsQL.md): allow overriding built-in function names inside [WITH templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs). For example, `WITH (sum(a,b) = a + b + 1) sum(x,y)` now expands into `x + y + 1`. Previously such a query would fail with `cannot use reserved name` error. See [this bugreport](https://github.com/VictoriaMetrics/metricsql/issues/5). -* BUGFIX: [vmui](./README.md#vmui): properly display values greater than 1000 on Y axis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2409). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly propagate limits at `-search.max*` command-line flags from `vminsert` to `vmstorage`. The limits are `-search.maxUniqueTimeseries`, `-search.maxSeries`, `-search.maxFederateSeries`, `-search.maxExportSeries`, `-search.maxGraphiteSeries` and `-search.maxTSDBStatusSeries`. They weren't propagated to `vmstorage` because of the bug. These limits were introduced in [v1.76.0](https://docs.victoriametrics.com/changelog/#v1760). See [this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2450). +* BUGFIX: fix goroutine leak and possible deadlock when importing invalid data via [native binary format](https://docs.victoriametrics.com/#how-to-import-data-in-native-format). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2423). +* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): properly calculate [hitCount](https://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.hitcount) function. Previously it could return empty results if there were no original samples in some parts of the selected time range. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow overriding built-in function names inside [WITH templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs). For example, `WITH (sum(a,b) = a + b + 1) sum(x,y)` now expands into `x + y + 1`. Previously such a query would fail with `cannot use reserved name` error. See [this bugreport](https://github.com/VictoriaMetrics/metricsql/issues/5). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly display values greater than 1000 on Y axis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2409). ## [v1.76.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.76.0) @@ -822,27 +822,27 @@ Released at 2022-04-07 **Update notes:** this release introduces backwards-incompatible changes to communication protocol between `vmselect` and `vmstorage` nodes in cluster version of VictoriaMetrics, so read requests to `vmselect` will fail until the upgrade is complete. These errors will stop after all the `vmselect` and `vmstorage` nodes are updated to the new release. It is safe to downgrade to previous releases. -* FEATURE: [vmctl](./vmctl.md): add ability to verify files obtained via [native export](./README.md#how-to-export-data-in-native-format). See [these docs](./vmctl.md#verifying-exported-blocks-from-victoriametrics) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362). -* FEATURE: [vmui](./README.md#vmui): add pre-defined dashboards for per-job CPU usage, memory usage and disk IO usage. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2243) for details. -* FEATURE: [vmalert](./vmalert.md): improve compatibility with [Prometheus Alert Generator specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2340). -* FEATURE: [vmalert](./vmalert.md): add `-datasource.disableKeepAlive` command-line flag, which can be used for disabling [HTTP keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection) to datasources. This option can be useful for distributing load among multiple datasources behind TCP proxy such as [HAProxy](http://www.haproxy.org/). -* FEATURE: [Cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md): reduce memory usage by up to 50% for `vminsert` and `vmstorage` under high ingestion rate. -* FEATURE: [vmgateway](./vmgateway.md): Allow to read `-ratelimit.config` file from URL. Also add `-ratelimit.configCheckInterval` command-line option. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2241). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add ability to verify files obtained via [native export](https://docs.victoriametrics.com/#how-to-export-data-in-native-format). See [these docs](https://docs.victoriametrics.com/vmctl/#verifying-exported-blocks-from-victoriametrics) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add pre-defined dashboards for per-job CPU usage, memory usage and disk IO usage. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2243) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): improve compatibility with [Prometheus Alert Generator specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2340). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-datasource.disableKeepAlive` command-line flag, which can be used for disabling [HTTP keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection) to datasources. This option can be useful for distributing load among multiple datasources behind TCP proxy such as [HAProxy](http://www.haproxy.org/). +* FEATURE: [Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/): reduce memory usage by up to 50% for `vminsert` and `vmstorage` under high ingestion rate. +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): Allow to read `-ratelimit.config` file from URL. Also add `-ratelimit.configCheckInterval` command-line option. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2241). * FEATURE: add the following command-line flags, which can be used for fine-grained limiting of CPU and memory usage during various API calls: - * `-search.maxFederateSeries` for limiting the number of time series, which can be returned from [/federate](./README.md#federation). - * `-search.maxExportSeries` for limiting the number of time series, which can be returned from [/api/v1/export](./README.md#how-to-export-time-series). - * `-search.maxSeries` for limiting the number of time series, which can be returned from [/api/v1/series](./url-examples.md#apiv1series). - * `-search.maxTSDBStatusSeries` for limiting the number of time series, which can be scanned during the request to [/api/v1/status/tsdb](./README.md#tsdb-stats). - * `-search.maxGraphiteSeries` for limiting the number of time series, which can be scanned during the request to [Graphite Render API](./README.md#graphite-render-api-usage). + * `-search.maxFederateSeries` for limiting the number of time series, which can be returned from [/federate](https://docs.victoriametrics.com/#federation). + * `-search.maxExportSeries` for limiting the number of time series, which can be returned from [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-time-series). + * `-search.maxSeries` for limiting the number of time series, which can be returned from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). + * `-search.maxTSDBStatusSeries` for limiting the number of time series, which can be scanned during the request to [/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats). + * `-search.maxGraphiteSeries` for limiting the number of time series, which can be scanned during the request to [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). -Previously the `-search.maxUniqueTimeseries` command-line flag was used as a global limit for all these APIs. Now the `-search.maxUniqueTimeseries` is used only for limiting the number of time series, which can be scanned during requests to [/api/v1/query](./url-examples.md#apiv1query) and [/api/v1/query_range](./url-examples.md#apiv1query_range). +Previously the `-search.maxUniqueTimeseries` command-line flag was used as a global limit for all these APIs. Now the `-search.maxUniqueTimeseries` is used only for limiting the number of time series, which can be scanned during requests to [/api/v1/query](https://docs.victoriametrics.com/url-examples/#apiv1query) and [/api/v1/query_range](https://docs.victoriametrics.com/url-examples/#apiv1query_range). -When using [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md), these command-line flags (including `-search.maxUniqueTimeseries`) must be passed to `vmselect` instead of `vmstorage`. +When using [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/), these command-line flags (including `-search.maxUniqueTimeseries`) must be passed to `vmselect` instead of `vmstorage`. -* BUGFIX: [vmagent](./vmagent.md) and [vmauth](./vmauth.md): reduce the probability of `TLS handshake error from XX.XX.XX.XX: EOF` errors when `-remoteWrite.url` points to HTTPS url at `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/) and [vmauth](https://docs.victoriametrics.com/vmauth/): reduce the probability of `TLS handshake error from XX.XX.XX.XX: EOF` errors when `-remoteWrite.url` points to HTTPS url at `vmauth`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699). * BUGFIX: return `Content-Type: text/html` response header when requesting `/` HTTP path at VictoriaMetrics components. Previously `text/plain` response header was returned, which could lead to broken page formatting. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2323). -* BUGFIX: [Graphite Render API](./README.md#graphite-render-api-usage): accept floating-point values for [maxDataPoints](https://graphite.readthedocs.io/en/stable/render_api.html#maxdatapoints) query arg, since some clients send floating-point values instead of integer values for this arg. +* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): accept floating-point values for [maxDataPoints](https://graphite.readthedocs.io/en/stable/render_api.html#maxdatapoints) query arg, since some clients send floating-point values instead of integer values for this arg. ## [v1.75.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.75.1) @@ -859,20 +859,20 @@ It replaces the `api/v1/groups` API handler with `api/v1/rules` handler in order with [alerts generator specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md). See other changes introduced to vmalert [here](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2320). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add support for mTLS communications between cluster components. See [these docs](./Cluster-VictoriaMetrics.md#mtls-protection) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/550). -* FEATURE: [vmalert](./vmalert.md): add ability to use OAuth2 for `-datasource.url`, `-notifier.url` and `-remoteRead.url`. See the corresponding command-line flags containing `oauth2` in their names [here](./vmalert.md#flags). -* FEATURE: [vmalert](./vmalert.md): add ability to use Bearer Token for `-notifier.url` via `-notifier.bearerToken` and `-notifier.bearerTokenFile` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1824). -* FEATURE: [vmalert](./vmalert.md): add `sortByLabel` template function in order to be consistent with Prometheus. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/#functions) for more details. -* FEATURE: [vmalert](./vmalert.md): improve compliance with [Prometheus Alert Generator Specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md). -* FEATURE: [vmalert](./vmalert.md): add `-rule.resendDelay` command-line flag, which specifies the minumum amount of time to wait before resending an alert to Alertmanager (e.g. this is equivalent to `-rules.alert.resend-delay` option from Prometheus. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1665). -* FEATURE: [vmauth](./vmauth.md): transparently treat `Authorization: Token ...` request headers as `Authorization: Bearer ...` request headers. This allows sending requests to `vmauth` from InfluxDB clients. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1897). Thanks to @dcircelli for the pull request. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add support for mTLS communications between cluster components. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/550). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to use OAuth2 for `-datasource.url`, `-notifier.url` and `-remoteRead.url`. See the corresponding command-line flags containing `oauth2` in their names [here](https://docs.victoriametrics.com/vmalert/#flags). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to use Bearer Token for `-notifier.url` via `-notifier.bearerToken` and `-notifier.bearerTokenFile` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1824). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `sortByLabel` template function in order to be consistent with Prometheus. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/#functions) for more details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): improve compliance with [Prometheus Alert Generator Specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-rule.resendDelay` command-line flag, which specifies the minumum amount of time to wait before resending an alert to Alertmanager (e.g. this is equivalent to `-rules.alert.resend-delay` option from Prometheus. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1665). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): transparently treat `Authorization: Token ...` request headers as `Authorization: Bearer ...` request headers. This allows sending requests to `vmauth` from InfluxDB clients. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1897). Thanks to @dcircelli for the pull request. * FEATURE: do not log trivial network errors such as `broken pipe` and `connection reset by peer`. This error could occur when writing data to the client, which closes the connection to VictoriaMetrics due to request timeout or similar reason. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2334). -* BUGFIX: [Graphite Render API](./README.md#graphite-render-api-usage): return an additional point after `until` timestamp in the same way as Graphite does. Previously VictoriaMetrics didn't return this point, which could result in missing last point on the graph. +* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): return an additional point after `until` timestamp in the same way as Graphite does. Previously VictoriaMetrics didn't return this point, which could result in missing last point on the graph. * BUGFIX: properly locate series with the given `name` and without the given `label` when using the `name{label=~"foo|"}` series selector. Previously such series could be skipped. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2255). Thanks to @jduncan0000 for discovering and fixing the issue. * BUGFIX: properly free up memory occupied by deleted cache entries for the following caches: `indexdb/dataBlocks`, `indexdb/indexBlocks`, `storage/indexBlocks`. This should reduce the increased memory usage starting from v1.73.0. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007) issue. * BUGFIX: reduce the interval for checking for free disk space from 30 seconds to 1 second. This should reduce the probability of `no space left on device` panics when `-storage.minFreeDiskSpaceBytes` is set to too low values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2305). -* BUGFIX: [vmagent](./vmagent.md): prevent from panic at vmagent when importing a time series with big number of samples. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2335). Thanks to @bleedfish for discovering and fixing the issue. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent from panic at vmagent when importing a time series with big number of samples. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2335). Thanks to @bleedfish for discovering and fixing the issue. ## [v1.74.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.74.0) @@ -895,15 +895,15 @@ This rule is equivalent to less clear traditional one: regex: 'foo;baz' ``` - See [relabeling docs](./vmagent.md#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998) for more details. + See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998) for more details. -* FEATURE: reduce memory usage for various caches under [high churn rate](./FAQ.md#what-is-high-churn-rate). -* FEATURE: [vmagent](./vmagent.md): re-use Kafka client when pushing data from [many tenants](./vmagent.md#multitenancy) to Kafka. Previously a separate Kafka client was created per each tenant. This could lead to increased load on Kafka. See [how to push data from vmagent to Kafka](./vmagent.md#writing-metrics-to-kafka). +* FEATURE: reduce memory usage for various caches under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): re-use Kafka client when pushing data from [many tenants](https://docs.victoriametrics.com/vmagent/#multitenancy) to Kafka. Previously a separate Kafka client was created per each tenant. This could lead to increased load on Kafka. See [how to push data from vmagent to Kafka](https://docs.victoriametrics.com/vmagent/#writing-metrics-to-kafka). * FEATURE: improve performance when registering new time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247). Thanks to @ahfuzhang . -* BUGFIX: return the proper number of datapoints from `moving*()` functions such as `movingAverage()` in [Graphite Render API](./README.md#graphite-render-api-usage). Previously these functions could return too big number of samples if [maxDataPoints query arg](https://graphite.readthedocs.io/en/stable/render_api.html#maxdatapoints) is explicitly passed to `/render` API. +* BUGFIX: return the proper number of datapoints from `moving*()` functions such as `movingAverage()` in [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously these functions could return too big number of samples if [maxDataPoints query arg](https://graphite.readthedocs.io/en/stable/render_api.html#maxdatapoints) is explicitly passed to `/render` API. * BUGFIX: properly handle [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) containing a filter for multiple metric names plus a negative filter. For example, `{__name__=~"foo|bar",job!="baz"}` . Previously VictoriaMetrics could return series with `foo` or `bar` names and with `job="baz"`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238). -* BUGFIX: [vmgateway](./vmgateway.md): properly parse JWT tokens if they are encoded with [URL-safe base64 encoding](https://datatracker.ietf.org/doc/html/rfc4648#section-5). +* BUGFIX: [vmgateway](https://docs.victoriametrics.com/vmgateway/): properly parse JWT tokens if they are encoded with [URL-safe base64 encoding](https://datatracker.ietf.org/doc/html/rfc4648#section-5). ## [v1.73.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.1) @@ -915,16 +915,16 @@ Released at 2022-02-22 * `storage/tsid` - the cache speeds up lookups of internal metric ids by `metric_name{labels...}` during data ingestion. The size for this cache can be tuned with `-storage.cacheSizeStorageTSID` command-line flag. * `indexdb/dataBlocks` - the cache speeds up data lookups in `<-storageDataPath>/indexdb` files. The size for this cache can be tuned with `-storage.cacheSizeIndexDBDataBlocks` command-line flag. * `indexdb/indexBlocks` - the cache speeds up index lookups in `<-storageDataPath>/indexdb` files. The size for this cache can be tuned with `-storage.cacheSizeIndexDBIndexBlocks` command-line flag. - See also [cache tuning docs](./README.md#cache-tuning). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940). -* FEATURE: add `-influxDBLabel` command-line flag for overriding `db` label name for the data [imported into VictoriaMetrics via InfluxDB line protocol](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). Thanks to @johnatannvmd for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2203). -* FEATURE: return `X-Influxdb-Version` HTTP header in responses to [InfluxDB write requests](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). This is needed for some InfluxDB clients. See [this comment](https://github.com/ntop/ntopng/issues/5449#issuecomment-1005347597) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2209). + See also [cache tuning docs](https://docs.victoriametrics.com/#cache-tuning). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940). +* FEATURE: add `-influxDBLabel` command-line flag for overriding `db` label name for the data [imported into VictoriaMetrics via InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). Thanks to @johnatannvmd for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2203). +* FEATURE: return `X-Influxdb-Version` HTTP header in responses to [InfluxDB write requests](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf). This is needed for some InfluxDB clients. See [this comment](https://github.com/ntop/ntopng/issues/5449#issuecomment-1005347597) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2209). * BUGFIX: reduce memory usage during the first three hours after the upgrade from versions older than v1.73.0. The memory usage spike was related to the need of in-memory caches' re-population after the upgrade because of the fix for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401). Now cache size limits are reduced in order to occupy less memory during the upgrade. * BUGFIX: fix a bug, which could significantly slow down requests to `/api/v1/labels` and `/api/v1/label//values`. These APIs are used by Grafana for auto-completion of label names and label values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200). * BUGFIX: vmalert: add support for `$externalLabels` and `$externalURL` template vars in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2193). * BUGFIX: vmalert: make sure notifiers are discovered during initialization if they are configured via `consul_sd_configs`. Previously they could be discovered in 30 seconds (the default value for `-promscrape.consulSDCheckInterval` command-line flag) after the initialization. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2202). * BUGFIX: update default value for `-promscrape.fileSDCheckInterval`, so it matches default duration used by Prometheus for checking for updates in `file_sd_configs`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2187). Thanks to @corporate-gadfly for the fix. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): do not return partial responses from `vmselect` if at least a single `vmstorage` node was reachable and returned an app-level error. Such errors are usually related to cluster mis-configuration, so they must be returned to the caller instead of being masked by [partial responses](./Cluster-VictoriaMetrics.md#cluster-availability). Partial responses can be returned only if some of `vmstorage` nodes are unreachable during the query. This may help the following issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1941), [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/678). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): do not return partial responses from `vmselect` if at least a single `vmstorage` node was reachable and returned an app-level error. Such errors are usually related to cluster mis-configuration, so they must be returned to the caller instead of being masked by [partial responses](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability). Partial responses can be returned only if some of `vmstorage` nodes are unreachable during the query. This may help the following issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1941), [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/678). ## [v1.73.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.0) @@ -934,36 +934,36 @@ Released at 2022-02-14 * FEATURE: publish VictoriaMetrics binaries for MacOS amd64 and MacOS arm64 (aka MacBook M1) at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1896) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1851). * FEATURE: reduce CPU and disk IO usage during `indexdb` rotation once per `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `-dropSamplesOnOverload` command-line flag for `vminsert`. If this flag is set, then `vminsert` drops incoming data if the destination `vmstorage` is temporarily unavailable or cannot keep up with the ingestion rate. The number of dropped rows can be [monitored](./Cluster-VictoriaMetrics.md#monitoring) via `vm_rpc_rows_dropped_on_overload_total` metric at `vminsert`. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): improve re-routing logic, so it re-routes incoming data more evenly if some of `vmstorage` nodes are temporarily unavailable and/or accept data at slower rate than other `vmstorage` nodes. Also significantly reduce possible re-routing storm when `vminsert` runs with `-disableRerouting=false` command-line flag. This should help the following issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1337), [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165), [three](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054), [four](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791), [five](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1544). -* FEATURE: [MetricsQL](./MetricsQL.md): cover more cases with the [label filters' propagation optimization](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization). This should improve the average performance for practical queries. The following cases are additionally covered: - * Multi-level [transform functions](./MetricsQL.md#transform-functions). For example, `abs(round(foo{a="b"})) + bar{x="y"}` is now optimized to `abs(round(foo{a="b",x="y"})) + bar{a="b",x="y"}` +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `-dropSamplesOnOverload` command-line flag for `vminsert`. If this flag is set, then `vminsert` drops incoming data if the destination `vmstorage` is temporarily unavailable or cannot keep up with the ingestion rate. The number of dropped rows can be [monitored](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) via `vm_rpc_rows_dropped_on_overload_total` metric at `vminsert`. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): improve re-routing logic, so it re-routes incoming data more evenly if some of `vmstorage` nodes are temporarily unavailable and/or accept data at slower rate than other `vmstorage` nodes. Also significantly reduce possible re-routing storm when `vminsert` runs with `-disableRerouting=false` command-line flag. This should help the following issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1337), [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165), [three](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054), [four](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791), [five](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1544). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): cover more cases with the [label filters' propagation optimization](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization). This should improve the average performance for practical queries. The following cases are additionally covered: + * Multi-level [transform functions](https://docs.victoriametrics.com/metricsql/#transform-functions). For example, `abs(round(foo{a="b"})) + bar{x="y"}` is now optimized to `abs(round(foo{a="b",x="y"})) + bar{a="b",x="y"}` * Binary operations with `on()`, `without()`, `group_left()` and `group_right()` modifiers. For example, `foo{a="b"} on (a) + bar` is now optimized to `foo{a="b"} on (a) + bar{a="b"}` * Multi-level binary operations. For example, `foo{a="b"} + bar{x="y"} + baz{z="q"}` is now optimized to `foo{a="b",x="y",z="q"} + bar{a="b",x="y",z="q"} + baz{a="b",x="y",z="q"}` * Aggregate functions. For example, `sum(foo{a="b"}) by (c) + bar{c="d"}` is now optimized to `sum(foo{a="b",c="d"}) by (c) + bar{c="d"}` -* FEATURE [MetricsQL](./MetricsQL.md): optimize joining with `*_info` labels. For example: `kube_pod_created{namespace="prod"} * on (uid) group_left(node) kube_pod_info` now automatically adds the needed filters on `uid` label to `kube_pod_info` before selecting series for the right side of `*` operation. This may save CPU, RAM and disk IO resources. See [this article](https://www.robustperception.io/exposing-the-software-version-to-prometheus) for details on `*_info` labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827). +* FEATURE [MetricsQL](https://docs.victoriametrics.com/metricsql/): optimize joining with `*_info` labels. For example: `kube_pod_created{namespace="prod"} * on (uid) group_left(node) kube_pod_info` now automatically adds the needed filters on `uid` label to `kube_pod_info` before selecting series for the right side of `*` operation. This may save CPU, RAM and disk IO resources. See [this article](https://www.robustperception.io/exposing-the-software-version-to-prometheus) for details on `*_info` labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827). * FEATURE: all: improve performance for arm64 builds of VictoriaMetrics components by up to 15%. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2102). * FEATURE: all: expose `process_cpu_cores_available` metric, which shows the number of CPU cores available to the app. The number can be fractional if the corresponding cgroup limit is set to a fractional value. This metric is useful for alerting on CPU saturation. For example, the following query alerts when the app uses more than 90% of CPU during the last 5 minutes: `rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9` . See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107). -* FEATURE: [vmalert](./vmalert.md): add ability to configure notifiers (e.g. alertmanager) via a file in the way similar to Prometheus. See [these docs](./vmalert.md#notifier-configuration-file), [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2127). -* FEATURE: [vmalert](./vmalert.md): add support for Consul service discovery for notifiers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1947). -* FEATURE: [vmalert](./vmalert.md): add support for specifying Basic Auth password for notifiers via a file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1567). -* FEATURE: [vmagent](./vmagent.md): provide the ability to fetch target responses on behalf of `vmagent` by clicking the `response` link for the needed target at `/targets` page. This feature may be useful for debugging responses from targets located in isolated environments. -* FEATURE: [vmagent](./vmagent.md): show the total number of scrapes and the total number of scrape errors per target at `/targets` page. This information may be useful when debugging unreliable scrape targets. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to configure notifiers (e.g. alertmanager) via a file in the way similar to Prometheus. See [these docs](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file), [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2127). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add support for Consul service discovery for notifiers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1947). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add support for specifying Basic Auth password for notifiers via a file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1567). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): provide the ability to fetch target responses on behalf of `vmagent` by clicking the `response` link for the needed target at `/targets` page. This feature may be useful for debugging responses from targets located in isolated environments. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): show the total number of scrapes and the total number of scrape errors per target at `/targets` page. This information may be useful when debugging unreliable scrape targets. * FEATURE: vmagent and single-node VictoriaMetrics: disallow unknown fields at `-promscrape.config` file. Previously unknown fields were allowed. This could lead to long-living silent config errors. The previous behaviour can be returned by passing `-promscrape.config.strictParse=false` command-line flag. -* FEATURE: vmagent: add `__meta_kubernetes_endpointslice_label*` and `__meta_kubernetes_endpointslice_annotation*` labels for `role: endpointslice` targets in [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs) to be consistent with other `role` values. See [this issue](https://github.com/prometheus/prometheus/issues/10284). +* FEATURE: vmagent: add `__meta_kubernetes_endpointslice_label*` and `__meta_kubernetes_endpointslice_annotation*` labels for `role: endpointslice` targets in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) to be consistent with other `role` values. See [this issue](https://github.com/prometheus/prometheus/issues/10284). * FEATURE: vmagent: add `collapse all` and `expand all` buttons to `http://vmagent:8429/targets` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021). * FEATURE: vmagent: support Prometheus-like durations in `-promscrape.config`. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766). * FEATURE: automatically re-read `-tlsCertFile` and `-tlsKeyFile` files, so their contents can be updated without the need to restart VictoriaMetrics apps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171). -* BUGFIX: calculate [absent_over_time()](./MetricsQL.md#absent_over_time) in the same way as Prometheus does. Previously it could return multiple time series instead of at most one time series like Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130). -* BUGFIX: return proper results from `highestMax()` function at [Graphite render API](./README.md#graphite-render-api-usage). Previously it was incorrectly returning timeseries with min peaks instead of max peaks. +* BUGFIX: calculate [absent_over_time()](https://docs.victoriametrics.com/metricsql/#absent_over_time) in the same way as Prometheus does. Previously it could return multiple time series instead of at most one time series like Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130). +* BUGFIX: return proper results from `highestMax()` function at [Graphite render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously it was incorrectly returning timeseries with min peaks instead of max peaks. * BUGFIX: properly limit indexdb cache sizes. Previously they could exceed values set via `-memory.allowedPercent` and/or `-memory.allowedBytes` when `indexdb` contained many data parts. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007). -* BUGFIX: [vmui](./README.md#vmui): fix a bug, which could break time range picker when editing `From` or `To` input fields. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2080). -* BUGFIX: [vmui](./README.md#vmui): fix a bug, which could break switching between `graph`, `json` and `table` views. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2084). -* BUGFIX: [vmui](./README.md#vmui): fix possible UI freeze after querying `node_uname_info` time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2115). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a bug, which could break time range picker when editing `From` or `To` input fields. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2080). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a bug, which could break switching between `graph`, `json` and `table` views. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2084). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix possible UI freeze after querying `node_uname_info` time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2115). * BUGFIX: show the original location of the warning or error message when logging throttled messages. Previously the location inside `lib/logger/throttler.go` was shown. This could increase the complexity of debugging. * BUGFIX: vmalert: fix links at web UI. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2167). -* BUGFIX: vmagent: properly discover pods without exposed ports for the given service for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_config](./sd_configs.md#kubernetes_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134). +* BUGFIX: vmagent: properly discover pods without exposed ports for the given service for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134). * BUGFIX: vmagent: properly display `zone` contents for `gce_sd_configs` section at `http://vmagent:8429/config` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2179). Thanks to @artifactori for the bugfix. * BUGFIX: vmagent: properly handle `all_tenants: true` config option at `openstack_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182). @@ -971,11 +971,11 @@ Released at 2022-02-14 Released at 2022-01-18 -* FEATURE: [MetricsQL](./MetricsQL.md): add support for `@` modifier, which is enabled by default in Prometheus starting from [Prometheus v2.33.0](https://github.com/prometheus/prometheus/pull/10121). See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348). VictoriaMetrics extends `@` modifier with the following additional features: +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add support for `@` modifier, which is enabled by default in Prometheus starting from [Prometheus v2.33.0](https://github.com/prometheus/prometheus/pull/10121). See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348). VictoriaMetrics extends `@` modifier with the following additional features: * It can contain arbitrary expression. For example, `foo @ (end() - 1h)` would return `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`. Another example: `foo @ (now() - 10m)` would return `foo` value 10 minutes ago from the current time. * It can be put everywhere in the query. For example, `sum(foo) @ start()` would calculate `sum(foo)` at `start` timestamp on the selected time range `[start ... end]`. -* FEATURE: [MetricsQL](./MetricsQL.md): add support for optional `keep_metric_names` modifier, which can be applied to all the [rollup functions](./MetricsQL.md#rollup-functions) and [transform functions](./MetricsQL.md#transform-functions). This modifier prevents from deleting metric names from function results. For example, `rate({__name__=~"foo|bar"}[5m]) keep_metric_names` leaves `foo` and `bar` metric names in `rate()` results. This feature provides an additional workaround for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949). -* FEATURE: [vmagent](./vmagent.md): add support for Kubernetes service discovery in the current namespace in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9881). For example, the following config limits pod discovery to the namespace where vmagent runs: +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add support for optional `keep_metric_names` modifier, which can be applied to all the [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) and [transform functions](https://docs.victoriametrics.com/metricsql/#transform-functions). This modifier prevents from deleting metric names from function results. For example, `rate({__name__=~"foo|bar"}[5m]) keep_metric_names` leaves `foo` and `bar` metric names in `rate()` results. This feature provides an additional workaround for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for Kubernetes service discovery in the current namespace in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9881). For example, the following config limits pod discovery to the namespace where vmagent runs: ```yaml scrape_configs: @@ -986,28 +986,28 @@ Released at 2022-01-18 own_namespace: true ``` -* FEATURE: [vmagent](./vmagent.md): add `__meta_kubernetes_node_provider_id` label for discovered Kubernetes nodes in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9603). -* FEATURE: [vmagent](./vmagent.md): log error message when remote storage returns 400 or 409 http errors. This should simplify detection and debugging of this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911). -* FEATURE: [vmagent](./vmagent.md): expose `promscrape_stale_samples_created_total` metric for monitoring the total number of created stale samples when scraping Prometheus targets. See [these docs](./vmagent.md#prometheus-staleness-markers) for the information on when stale samples (aka staleness markers) can be created. -* FEATURE: [vmrestore](./vmrestore.md): store `restore-in-progress` file in `-dst` directory while `vmrestore` is running. This file is automatically deleted when `vmrestore` is successfully finished. This helps detecting incompletely restored data on VictoriaMetrics start. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958). -* FEATURE: [vmctl](./vmctl.md): print the last sample timestamp when the data migration is interrupted either by user or by error. This helps continuing the data migration from the interruption moment. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1236). -* FEATURE: [vmalert](./vmalert.md): expose `vmalert_remotewrite_total` metric at `/metrics` page. This makes possible calculating SLOs for error rate during writing recording rules and alert state to `-remoteWrite.url` with the query `vmalert_remotewrite_errors_total / vmalert_remotewrite_total`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2040). Thanks to @afoninsky . -* FEATURE: [vmalert](./vmalert.md): add `stripPort` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/10002). -* FEATURE: [vmalert](./vmalert.md): add `parseDuration` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/8817). -* FEATURE: [MetricsQL](./MetricsQL.md): add `stale_samples_over_time(m[d])` function for calculating the number of [staleness marks](./vmagent.md#prometheus-staleness-markers) for time series `m` over the duration `d`. This function may be useful for detecting flapping metrics at scrape targets, which periodically disappear and then appear again. -* FEATURE: [vmgateway](./vmgateway.md): add support for `extra_filters` option. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863). -* FEATURE: [vmui](./README.md#vmui): improve UX according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960). Thanks to @Loori-R . -* FEATURE: [vmui](./README.md#vmui): limit the number of requests sent to VictoriaMetrics during zooming / scrolling. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_node_provider_id` label for discovered Kubernetes nodes in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9603). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): log error message when remote storage returns 400 or 409 http errors. This should simplify detection and debugging of this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose `promscrape_stale_samples_created_total` metric for monitoring the total number of created stale samples when scraping Prometheus targets. See [these docs](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for the information on when stale samples (aka staleness markers) can be created. +* FEATURE: [vmrestore](https://docs.victoriametrics.com/vmrestore/): store `restore-in-progress` file in `-dst` directory while `vmrestore` is running. This file is automatically deleted when `vmrestore` is successfully finished. This helps detecting incompletely restored data on VictoriaMetrics start. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): print the last sample timestamp when the data migration is interrupted either by user or by error. This helps continuing the data migration from the interruption moment. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1236). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): expose `vmalert_remotewrite_total` metric at `/metrics` page. This makes possible calculating SLOs for error rate during writing recording rules and alert state to `-remoteWrite.url` with the query `vmalert_remotewrite_errors_total / vmalert_remotewrite_total`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2040). Thanks to @afoninsky . +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `stripPort` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/10002). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `parseDuration` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/8817). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `stale_samples_over_time(m[d])` function for calculating the number of [staleness marks](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for time series `m` over the duration `d`. This function may be useful for detecting flapping metrics at scrape targets, which periodically disappear and then appear again. +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add support for `extra_filters` option. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve UX according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960). Thanks to @Loori-R . +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): limit the number of requests sent to VictoriaMetrics during zooming / scrolling. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064). -* BUGFIX: [vmagent](./vmagent.md): make sure that `vmagent` replicas scrape the same targets at different time offsets when [replication is enabled in vmagent clustering mode](./vmagent.md#scraping-big-number-of-targets). This guarantees that the [deduplication](./README.md#deduplication) consistently leaves samples from the same `vmagent` replica. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): make sure that `vmagent` replicas scrape the same targets at different time offsets when [replication is enabled in vmagent clustering mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). This guarantees that the [deduplication](https://docs.victoriametrics.com/#deduplication) consistently leaves samples from the same `vmagent` replica. * BUGFIX: return the proper response stub from `/api/v1/query_exemplars` handler, which is needed for Grafana v8+. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999). -* BUGFIX: [vmctl](./vmctl.md): fix a few edge cases and improve migration speed for OpenTSDB importer. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2019). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix a few edge cases and improve migration speed for OpenTSDB importer. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2019). * BUGFIX: fix possible data race when searching for time series matching `{key=~"value|"}` filter over time range covering multiple days. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2032). Thanks to @waldoweng for the provided fix. -* BUGFIX: [vmagent](./vmagent.md): do not send staleness markers on graceful shutdown. This follows Prometheus behavior. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079). -* BUGFIX: [vmagent](./vmagent.md): properly set `__address__` label in `dockerswarm_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2038). Thanks to @ashtuchkin for the fix. -* BUGFIX: [vmui](./README.md#vmui): fix incorrect calculations for graph limits on y axis. This could result in incorrect graph rendering in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037). -* BUGFIX: [vmui](./README.md#vmui): fix handling for multi-line queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not send staleness markers on graceful shutdown. This follows Prometheus behavior. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set `__address__` label in `dockerswarm_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2038). Thanks to @ashtuchkin for the fix. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix incorrect calculations for graph limits on y axis. This could result in incorrect graph rendering in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling for multi-line queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039). ## Previous releases -See changes for older releases [here](./CHANGELOG_2021.md). +See changes for older releases [here](https://docs.victoriametrics.com/changelog_2021/). diff --git a/docs/CHANGELOG_2023.md b/docs/CHANGELOG_2023.md index a7af56a98..729134862 100644 --- a/docs/CHANGELOG_2023.md +++ b/docs/CHANGELOG_2023.md @@ -18,43 +18,43 @@ Released at 2023-12-13 * SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See [alpine 3.19.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html). * SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See [the list of issues addressed in Go1.21.5](https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved). -* FEATURE: [vmauth](./vmauth.md): add ability to send requests to the first available backend and fall back to other `hot standby` backends when the first backend is unavailable. This allows building highly available setups as shown in [these docs](./vmauth.md#high-availability). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792). -* FEATURE: `vmselect`: allow specifying multiple groups of `vmstorage` nodes with independent `-replicationFactor` per each group. See [these docs](./Cluster-VictoriaMetrics.md#vmstorage-groups-at-vmselect) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197) for details. -* FEATURE: `vmselect`: allow opening [vmui](./README.md#vmui) and investigating [Top queries](./README.md#top-queries) and [Active queries](./README.md#active-queries) when the `vmselect` is overloaded with concurrent queries (e.g. when more than `-search.maxConcurrentRequests` concurrent queries are executed). Previously an attempt to open `Top queries` or `Active queries` at `vmui` could result in `couldn't start executing the request in ... seconds, since -search.maxConcurrentRequests=... concurrent requests are executed` error, which could complicate debugging of overloaded `vmselect` or single-node VictoriaMetrics. -* FEATURE: [vmagent](./vmagent.md): add `-enableMultitenantHandlers` command-line flag, which allows receiving data via [VictoriaMetrics cluster urls](./Cluster-VictoriaMetrics.md#url-format) at `vmagent` and converting [tenant ids](./Cluster-VictoriaMetrics.md#multitenancy) to (`vm_account_id`, `vm_project_id`) labels before sending the data to the configured `-remoteWrite.url`. See [these docs](./vmagent.md#multitenancy) for details. -* FEATURE: [vmagent](./vmagent.md): add `-remoteWrite.disableOnDiskQueue` command-line flag, which can be used for disabling data queueing to disk when the remote storage cannot keep up with the data ingestion rate. See [these docs](./vmagent.md#disabling-on-disk-persistence) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110). -* FEATURE: [vmagent](./vmagent.md): add support for reading and writing samples via [Google PubSub](https://cloud.google.com/pubsub). See [these docs](./vmagent.md#google-pubsub-integration). -* FEATURE: [vmagent](./vmagent.md): show all the dropped targets together with the reason why they are dropped at `http://vmagent:8429/service-discovery` page. Previously targets, which were dropped because of [target sharding](./vmagent.md#scraping-big-number-of-targets) weren't displayed on this page. This could complicate service discovery debugging. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018). -* FEATURE: reduce the default value for `-import.maxLineLen` command-line flag from 100MB to 10MB in order to prevent excessive memory usage during data import via [/api/v1/import](./README.md#how-to-import-data-in-json-line-format). -* FEATURE: [vmagent](./vmagent.md): add `keep_if_contains` and `drop_if_contains` relabeling actions. See [these docs](./vmagent.md#relabeling-enhancements) for details. -* FEATURE: [vmagent](./vmagent.md): export `vm_promscrape_scrape_pool_targets` [metric](./vmagent.md#monitoring) to track the number of targets each scrape job discovers. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5311). -* FEATURE: [vmalert](./vmalert.md): provide `/vmalert/api/v1/rule` and `/api/v1/rule` API endpoints to get the rule object in JSON format. See [these docs](./vmalert.md#web) for details. -* FEATURE: [vmalert](./vmalert.md): deprecate process gauge metrics `vmalert_alerting_rules_error` and `vmalert_recording_rules_error` in favour of `vmalert_alerting_rules_errors_total` and `vmalert_recording_rules_errors_total` counter metrics. [Counter](./keyConcepts.md#counter) metric type is more suitable for error counting as it preserves the state change between the scrapes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5160) for details. -* FEATURE: [MetricsQL](./MetricsQL.md): add [day_of_year()](./MetricsQL.md#day_of_year) function, which returns the day of the year for each of the given unix timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5345) for details. Thanks to @luckyxiaoqiang for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5368/). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to send requests to the first available backend and fall back to other `hot standby` backends when the first backend is unavailable. This allows building highly available setups as shown in [these docs](https://docs.victoriametrics.com/vmauth/#high-availability). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792). +* FEATURE: `vmselect`: allow specifying multiple groups of `vmstorage` nodes with independent `-replicationFactor` per each group. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197) for details. +* FEATURE: `vmselect`: allow opening [vmui](https://docs.victoriametrics.com/#vmui) and investigating [Top queries](https://docs.victoriametrics.com/#top-queries) and [Active queries](https://docs.victoriametrics.com/#active-queries) when the `vmselect` is overloaded with concurrent queries (e.g. when more than `-search.maxConcurrentRequests` concurrent queries are executed). Previously an attempt to open `Top queries` or `Active queries` at `vmui` could result in `couldn't start executing the request in ... seconds, since -search.maxConcurrentRequests=... concurrent requests are executed` error, which could complicate debugging of overloaded `vmselect` or single-node VictoriaMetrics. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-enableMultitenantHandlers` command-line flag, which allows receiving data via [VictoriaMetrics cluster urls](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format) at `vmagent` and converting [tenant ids](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) to (`vm_account_id`, `vm_project_id`) labels before sending the data to the configured `-remoteWrite.url`. See [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-remoteWrite.disableOnDiskQueue` command-line flag, which can be used for disabling data queueing to disk when the remote storage cannot keep up with the data ingestion rate. See [these docs](https://docs.victoriametrics.com/vmagent/#disabling-on-disk-persistence) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for reading and writing samples via [Google PubSub](https://cloud.google.com/pubsub). See [these docs](https://docs.victoriametrics.com/vmagent/#google-pubsub-integration). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): show all the dropped targets together with the reason why they are dropped at `http://vmagent:8429/service-discovery` page. Previously targets, which were dropped because of [target sharding](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) weren't displayed on this page. This could complicate service discovery debugging. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018). +* FEATURE: reduce the default value for `-import.maxLineLen` command-line flag from 100MB to 10MB in order to prevent excessive memory usage during data import via [/api/v1/import](https://docs.victoriametrics.com/#how-to-import-data-in-json-line-format). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `keep_if_contains` and `drop_if_contains` relabeling actions. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): export `vm_promscrape_scrape_pool_targets` [metric](https://docs.victoriametrics.com/vmagent/#monitoring) to track the number of targets each scrape job discovers. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5311). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): provide `/vmalert/api/v1/rule` and `/api/v1/rule` API endpoints to get the rule object in JSON format. See [these docs](https://docs.victoriametrics.com/vmalert/#web) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): deprecate process gauge metrics `vmalert_alerting_rules_error` and `vmalert_recording_rules_error` in favour of `vmalert_alerting_rules_errors_total` and `vmalert_recording_rules_errors_total` counter metrics. [Counter](https://docs.victoriametrics.com/keyconcepts/#counter) metric type is more suitable for error counting as it preserves the state change between the scrapes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5160) for details. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [day_of_year()](https://docs.victoriametrics.com/metricsql/#day_of_year) function, which returns the day of the year for each of the given unix timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5345) for details. Thanks to @luckyxiaoqiang for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5368/). * FEATURE: all VictoriaMetrics binaries: expose additional metrics at `/metrics` page, which may simplify debugging of VictoriaMetrics components (see [this feature request](https://github.com/VictoriaMetrics/metrics/issues/54)): - * `go_sched_latencies_seconds` - the [histogram](./keyConcepts.md#histogram), which shows the time goroutines have spent in runnable state before actually running. Big values point to the lack of CPU time for the current workload. - * `go_mutex_wait_seconds_total` - the [counter](./keyConcepts.md#counter), which shows the total time spent by goroutines waiting for locked mutex. Big values point to mutex contention issues. - * `go_gc_cpu_seconds_total` - the [counter](./keyConcepts.md#counter), which shows the total CPU time spent by Go garbage collector. - * `go_gc_mark_assist_cpu_seconds_total` - the [counter](./keyConcepts.md#counter), which shows the total CPU time spent by goroutines in GC mark assist state. - * `go_gc_pauses_seconds` - the [histogram](./keyConcepts.md#histogram), which shows the duration of GC pauses. - * `go_scavenge_cpu_seconds_total` - the [counter](./keyConcepts.md#counter), which shows the total CPU time spent by Go runtime for returning memory to the Operating System. + * `go_sched_latencies_seconds` - the [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram), which shows the time goroutines have spent in runnable state before actually running. Big values point to the lack of CPU time for the current workload. + * `go_mutex_wait_seconds_total` - the [counter](https://docs.victoriametrics.com/keyconcepts/#counter), which shows the total time spent by goroutines waiting for locked mutex. Big values point to mutex contention issues. + * `go_gc_cpu_seconds_total` - the [counter](https://docs.victoriametrics.com/keyconcepts/#counter), which shows the total CPU time spent by Go garbage collector. + * `go_gc_mark_assist_cpu_seconds_total` - the [counter](https://docs.victoriametrics.com/keyconcepts/#counter), which shows the total CPU time spent by goroutines in GC mark assist state. + * `go_gc_pauses_seconds` - the [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram), which shows the duration of GC pauses. + * `go_scavenge_cpu_seconds_total` - the [counter](https://docs.victoriametrics.com/keyconcepts/#counter), which shows the total CPU time spent by Go runtime for returning memory to the Operating System. * `go_memlimit_bytes` - the value of [GOMEMLIMIT](https://pkg.go.dev/runtime#hdr-Environment_Variables) environment variable. -* FEATURE: [vmui](./README.md#vmui): enhance autocomplete functionality with caching. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5348). -* FEATURE: add field `version` to the response for `/api/v1/status/buildinfo` API for using more efficient API in Grafana for receiving label values. Add additional info about setup Grafana datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5370) and [these docs](./README.md#grafana-setup) for details. -* FEATURE: add `-search.maxResponseSeries` command-line flag for limiting the number of time series a single query to [`/api/v1/query`](./keyConcepts.md#instant-query) or [`/api/v1/query_range`](./keyConcepts.md#range-query) can return. This limit can protect Grafana from high memory usage when the query returns too many series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5372). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): enhance autocomplete functionality with caching. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5348). +* FEATURE: add field `version` to the response for `/api/v1/status/buildinfo` API for using more efficient API in Grafana for receiving label values. Add additional info about setup Grafana datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5370) and [these docs](https://docs.victoriametrics.com/#grafana-setup) for details. +* FEATURE: add `-search.maxResponseSeries` command-line flag for limiting the number of time series a single query to [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) or [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query) can return. This limit can protect Grafana from high memory usage when the query returns too many series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5372). * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): ease aggregation for certain alerting rules to keep more useful labels for the context. Before, all extra labels except `job` and `instance` were ignored. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5429) and this [follow-up commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/8fb68152e67712ed2c16dcfccf7cf4d0af140835). Thanks to @7840vz. -* FEATURE: [vmctl](./vmctl.md): allow reversing the migrating order from the newest to the oldest data for [vm-native](./vmctl.md#migrating-data-from-victoriametrics) and [remote-read](./vmctl.md#migrating-data-by-remote-read-protocol) modes via `--vm-native-filter-time-reverse` and `--remote-read-filter-time-reverse` command-line flags respectively. See: ./vmctl.md#using-time-based-chunking-of-migration and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5376). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): allow reversing the migrating order from the newest to the oldest data for [vm-native](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics) and [remote-read](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol) modes via `--vm-native-filter-time-reverse` and `--remote-read-filter-time-reverse` command-line flags respectively. See: https://docs.victoriametrics.com/vmctl/#using-time-based-chunking-of-migration and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5376). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly calculate values for the first point on the graph for queries, which do not use [rollup functions](./MetricsQL.md#rollup-functions). For example, previously `count(up)` could return lower than expected values for the first point on the graph. This also could result in lower than expected values in the middle of the graph like in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5388) when the response caching isn't disabled. The issue has been introduced in [v1.95.0](./CHANGELOG.md#v1950). -* BUGFIX: [vmagent](./vmagent.md): prevent from `FATAL: cannot flush metainfo` panic when [`-remoteWrite.multitenantURL`](./vmagent.md#multitenancy) command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357). -* BUGFIX: [vmagent](./vmagent.md): properly decode zstd-encoded data blocks received via [VictoriaMetrics remote_write protocol](./vmagent.md#victoriametrics-remote-write-protocol). See [this issue comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301#issuecomment-1815871992). -* BUGFIX: [vmagent](./vmagent.md): properly add new labels at `output_relabel_configs` during [stream aggregation](./stream-aggregation.md). Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing [detailed report for this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5402). -* BUGFIX: [vmalert-tool](./vmalert-tool.md): allow using arbitrary `eval_time` in [alert_rule_test](./vmalert-tool.md#alert_test_case) case. Previously, test cases with `eval_time` not being a multiple of `evaluation_interval` would fail. -* BUGFIX: [vmalert](./vmalert.md): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix `vmbackupmanager` not deleting previous object versions from S3 when applying retention policy with `-deleteAllObjectVersions` command-line flag. -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): fix panic when ingesting data via [NewRelic protocol](./README.md#how-to-send-data-from-newrelic-agent) into VictoriaMetrics cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5416). -* BUGFIX: properly escape `<` character in responses returned via [`/federate`](./README.md#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). -* BUGFIX: [vmctl](./vmctl.md): check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5446). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly calculate values for the first point on the graph for queries, which do not use [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions). For example, previously `count(up)` could return lower than expected values for the first point on the graph. This also could result in lower than expected values in the middle of the graph like in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5388) when the response caching isn't disabled. The issue has been introduced in [v1.95.0](https://docs.victoriametrics.com/changelog/#v1950). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent from `FATAL: cannot flush metainfo` panic when [`-remoteWrite.multitenantURL`](https://docs.victoriametrics.com/vmagent/#multitenancy) command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly decode zstd-encoded data blocks received via [VictoriaMetrics remote_write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). See [this issue comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301#issuecomment-1815871992). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly add new labels at `output_relabel_configs` during [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/). Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing [detailed report for this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5402). +* BUGFIX: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool.html): allow using arbitrary `eval_time` in [alert_rule_test](https://docs.victoriametrics.com/vmalert-tool.html#alert_test_case) case. Previously, test cases with `eval_time` not being a multiple of `evaluation_interval` would fail. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix `vmbackupmanager` not deleting previous object versions from S3 when applying retention policy with `-deleteAllObjectVersions` command-line flag. +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): fix panic when ingesting data via [NewRelic protocol](https://docs.victoriametrics.com/#how-to-send-data-from-newrelic-agent) into VictoriaMetrics cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5416). +* BUGFIX: properly escape `<` character in responses returned via [`/federate`](https://docs.victoriametrics.com/#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5446). ## [v1.95.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1) @@ -62,80 +62,80 @@ Released at 2023-11-16 * FEATURE: dashboards: use `version` instead of `short_version` in version change annotation for single/cluster dashboards. The update should reflect version changes even if different flavours of the same release were applied (custom builds). -* BUGFIX: fix a bug, which could result in improper results and/or to `cannot merge series: duplicate series found` error during [range query](./keyConcepts.md#range-query) execution. The issue has been introduced in [v1.95.0](./CHANGELOG.md#v1950). See [this bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332) for details. +* BUGFIX: fix a bug, which could result in improper results and/or to `cannot merge series: duplicate series found` error during [range query](https://docs.victoriametrics.com/keyconcepts/#range-query) execution. The issue has been introduced in [v1.95.0](https://docs.victoriametrics.com/changelog/#v1950). See [this bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332) for details. * BUGFIX: improve deadline detection when using buffered connection for communication between cluster components. Before, due to nature of a buffered connection the deadline could have been exceeded while reading or writing buffered data to connection. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5327). ## [v1.95.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.0) Released at 2023-11-15 -**It is recommended upgrading to [v1.95.1](./CHANGELOG.md#v1951) because v1.95.0 contains a bug, which can lead to incorrect query results and to `cannot merge series: duplicate series found` error. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332) for details.** +**It is recommended upgrading to [v1.95.1](https://docs.victoriametrics.com/changelog/#v1951) because v1.95.0 contains a bug, which can lead to incorrect query results and to `cannot merge series: duplicate series found` error. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332) for details.** -**vmalert's cmd-line flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead and see more details on how to use it [here](./vmalert.md#data-delay). The flag `datasource.lookback` will have no effect in the next release and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).** +**vmalert's cmd-line flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead and see more details on how to use it [here](https://docs.victoriametrics.com/vmalert/#data-delay). The flag `datasource.lookback` will have no effect in the next release and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).** **vmalert's cmd-line flag `-datasource.queryTimeAlignment` was deprecated and will have no effect anymore. It will be completely removed in next releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049) and more detailed changes related to vmalert below.** * SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.4. See [the list of issues addressed in Go1.21.2](https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved), [the list of issues addressed in Go1.21.3](https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved) and [the list of issues addressed in Go1.21.4](https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved). -* FEATURE: `vmselect`: improve performance for repeated [instant queries](./keyConcepts.md#instant-query) if they contain one of the following [rollup functions](./MetricsQL.md#rollup-functions): - - [`avg_over_time`](./MetricsQL.md#avg_over_time) - - [`sum_over_time`](./MetricsQL.md#sum_over_time) - - [`count_eq_over_time`](./MetricsQL.md#count_eq_over_time) - - [`count_gt_over_time`](./MetricsQL.md#count_gt_over_time) - - [`count_le_over_time`](./MetricsQL.md#count_le_over_time) - - [`count_ne_over_time`](./MetricsQL.md#count_ne_over_time) - - [`count_over_time`](./MetricsQL.md#count_over_time) - - [`increase`](./MetricsQL.md#increase) - - [`max_over_time`](./MetricsQL.md#max_over_time) - - [`min_over_time`](./MetricsQL.md#min_over_time) - - [`rate`](./MetricsQL.md#rate) +* FEATURE: `vmselect`: improve performance for repeated [instant queries](https://docs.victoriametrics.com/keyconcepts/#instant-query) if they contain one of the following [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions): + - [`avg_over_time`](https://docs.victoriametrics.com/metricsql/#avg_over_time) + - [`sum_over_time`](https://docs.victoriametrics.com/metricsql/#sum_over_time) + - [`count_eq_over_time`](https://docs.victoriametrics.com/metricsql/#count_eq_over_time) + - [`count_gt_over_time`](https://docs.victoriametrics.com/metricsql/#count_gt_over_time) + - [`count_le_over_time`](https://docs.victoriametrics.com/metricsql/#count_le_over_time) + - [`count_ne_over_time`](https://docs.victoriametrics.com/metricsql/#count_ne_over_time) + - [`count_over_time`](https://docs.victoriametrics.com/metricsql/#count_over_time) + - [`increase`](https://docs.victoriametrics.com/metricsql/#increase) + - [`max_over_time`](https://docs.victoriametrics.com/metricsql/#max_over_time) + - [`min_over_time`](https://docs.victoriametrics.com/metricsql/#min_over_time) + - [`rate`](https://docs.victoriametrics.com/metricsql/#rate) The optimization is enabled when these functions contain lookbehind window in square brackets bigger or equal to `6h` (the threshold can be changed via `-search.minWindowForInstantRollupOptimization` command-line flag). The optimization improves performance for SLO/SLI-like queries such as `avg_over_time(up[30d])` or `sum(rate(http_request_errors_total[3d])) / sum(rate(http_requests_total[3d]))`, which can be generated by [sloth](https://github.com/slok/sloth) or similar projects. * FEATURE: `vmselect`: improve query performance on systems with big number of CPU cores (`>=32`). Add `-search.maxWorkersPerQuery` command-line flag, which can be used for fine-tuning query performance on systems with big number of CPU cores. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5195). * FEATURE: `vmselect`: expose `vm_memory_intensive_queries_total` counter metric which gets increased each time `-search.logQueryMemoryUsage` memory limit is exceeded by a query. This metric should help to identify expensive and heavy queries without inspecting the logs. -* FEATURE: [MetricsQL](./MetricsQL.md): add [drop_empty_series()](./MetricsQL.md#drop_empty_series) function, which can be used for filtering out empty series before performing additional calculations as shown in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071). -* FEATURE: [MetricsQL](./MetricsQL.md): add [labels_equal()](./MetricsQL.md#labels_equal) function, which can be used for searching series with identical values for the given labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148). -* FEATURE: [MetricsQL](./MetricsQL.md): add [`outlier_iqr_over_time(m[d])`](./MetricsQL.md#outlier_iqr_over_time) and [`outliers_iqr(q)`](./MetricsQL.md#outliers_iqr) functions, which allow detecting anomalies in samples and series using [Interquartile range method](https://en.wikipedia.org/wiki/Interquartile_range). -* FEATURE: [vmalert](./vmalert.md): add `eval_alignment` attribute for [Groups](./vmalert.md#groups), it will align group query requests timestamp with interval like `datasource.queryTimeAlignment` did. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [drop_empty_series()](https://docs.victoriametrics.com/metricsql/#drop_empty_series) function, which can be used for filtering out empty series before performing additional calculations as shown in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [labels_equal()](https://docs.victoriametrics.com/metricsql/#labels_equal) function, which can be used for searching series with identical values for the given labels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [`outlier_iqr_over_time(m[d])`](https://docs.victoriametrics.com/metricsql/#outlier_iqr_over_time) and [`outliers_iqr(q)`](https://docs.victoriametrics.com/metricsql/#outliers_iqr) functions, which allow detecting anomalies in samples and series using [Interquartile range method](https://en.wikipedia.org/wiki/Interquartile_range). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `eval_alignment` attribute for [Groups](https://docs.victoriametrics.com/vmalert/#groups), it will align group query requests timestamp with interval like `datasource.queryTimeAlignment` did. This also means that `datasource.queryTimeAlignment` command-line flag becomes deprecated now and will have no effect if configured. If `datasource.queryTimeAlignment` was set to `false` before, then `eval_alignment` has to be set to `false` explicitly under group. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049). -* FEATURE: [vmalert](./vmalert.md): add `-rule.evalDelay` flag and `eval_delay` attribute for [Groups](./vmalert.md#groups). The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match [intentional query delay](./keyConcepts.md#query-latency) from the datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). -* FEATURE: [vmalert](./vmalert.md): allow specifying full url in notifier static_configs target address, like `http://alertmanager:9093/test/api/v2/alerts`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5184). -* FEATURE: [vmalert](./vmalert.md): reduce the number of queries for restoring alerts state on start-up. The change should speed up the restore process and reduce pressure on `remoteRead.url`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5265). -* FEATURE: [vmalert](./vmalert.md): add label `file` pointing to the group's filename to metrics `vmalert_recording_.*` and `vmalert_alerts_.*`. The filename should help identifying alerting rules belonging to specific groups with identical names but different filenames. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267). -* FEATURE: [vmalert](./vmalert.md): automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers. -* FEATURE: [vmagent](./vmagent.md): support data ingestion from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-newrelic-agent), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712). -* FEATURE: [vmagent](./vmagent.md): add `-remoteWrite.shardByURL.labels` command-line flag, which can be used for specifying a list of labels for sharding outgoing samples among the configured `-remoteWrite.url` destinations if `-remoteWrite.shardByURL` command-line flag is set. See [these docs](./vmagent.md#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942) for details. -* FEATURE: [vmagent](./vmagent.md): do not exit on startup when [scrape_configs](./sd_configs.md#scrape_configs) refer to non-existing or invalid files with auth configs, since these files may appear / updated later. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). -* FEATURE: [vmagent](./vmagent.md): allow loading TLS certificates from HTTP and HTTPS urls by specifying these urls at `cert_file` and `key_file` options inside `tls_config` and `proxy_tls_config` sections at [http client settings](./sd_configs.md#http-api-client-options). -* FEATURE: [vmagent](./vmagent.md): reduce CPU load when big number of targets are scraped over HTTPS with the same custom TLS certificate configured via `tls_config->cert_file` and `tls_config->key_file` at [scrape_config](./sd_configs.md#scrape_configs). -* FEATURE: [vmbackup](./vmbackup.md): add `-filestream.disableFadvise` command-line flag, which can be used for disabling `fadvise` syscall during backup upload to the remote storage. By default `vmbackup` uses `fadvise` syscall in order to prevent from eviction of recently accessed data from the [OS page cache](https://en.wikipedia.org/wiki/Page_cache) when backing up large files. Sometimes the `fadvise` syscall may take significant amounts of CPU when the backup is performed with large value of `-concurrency` command-line flag on systems with big number of CPU cores. In this case it is better to manually disable `fadvise` syscall by passing `-filestream.disableFadvise` command-line flag to `vmbackup`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120) for details. -* FEATURE: [vmbackup](./vmbackup.md): add `-deleteAllObjectVersions` command-line flag, which can be used for forcing removal of all object versions in remote object storage. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121) issue and [these docs](./vmbackup.md#permanent-deletion-of-objects-in-s3-compatible-storages) for the details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-rule.evalDelay` flag and `eval_delay` attribute for [Groups](https://docs.victoriametrics.com/vmalert/#groups). The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match [intentional query delay](https://docs.victoriametrics.com/keyconcepts/#query-latency) from the datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow specifying full url in notifier static_configs target address, like `http://alertmanager:9093/test/api/v2/alerts`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5184). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): reduce the number of queries for restoring alerts state on start-up. The change should speed up the restore process and reduce pressure on `remoteRead.url`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5265). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add label `file` pointing to the group's filename to metrics `vmalert_recording_.*` and `vmalert_alerts_.*`. The filename should help identifying alerting rules belonging to specific groups with identical names but different filenames. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support data ingestion from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-newrelic-agent), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-remoteWrite.shardByURL.labels` command-line flag, which can be used for specifying a list of labels for sharding outgoing samples among the configured `-remoteWrite.url` destinations if `-remoteWrite.shardByURL` command-line flag is set. See [these docs](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): do not exit on startup when [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) refer to non-existing or invalid files with auth configs, since these files may appear / updated later. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow loading TLS certificates from HTTP and HTTPS urls by specifying these urls at `cert_file` and `key_file` options inside `tls_config` and `proxy_tls_config` sections at [http client settings](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU load when big number of targets are scraped over HTTPS with the same custom TLS certificate configured via `tls_config->cert_file` and `tls_config->key_file` at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): add `-filestream.disableFadvise` command-line flag, which can be used for disabling `fadvise` syscall during backup upload to the remote storage. By default `vmbackup` uses `fadvise` syscall in order to prevent from eviction of recently accessed data from the [OS page cache](https://en.wikipedia.org/wiki/Page_cache) when backing up large files. Sometimes the `fadvise` syscall may take significant amounts of CPU when the backup is performed with large value of `-concurrency` command-line flag on systems with big number of CPU cores. In this case it is better to manually disable `fadvise` syscall by passing `-filestream.disableFadvise` command-line flag to `vmbackup`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120) for details. +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): add `-deleteAllObjectVersions` command-line flag, which can be used for forcing removal of all object versions in remote object storage. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121) issue and [these docs](https://docs.victoriametrics.com/vmbackup/#permanent-deletion-of-objects-in-s3-compatible-storages) for the details. * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): account for `vmauth` component for alerts `ServiceDown` and `TooManyRestarts`. * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): make `TooHighMemoryUsage` more tolerable to spikes or near-the-threshold states. The change should reduce number of false positives. * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): add `TooManyMissedIterations` alerting rule for vmalert to detect groups that miss their evaluations due to slow queries. -* FEATURE: [vmui](./README.md#vmui): add support for functions, labels, values in autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006). -* FEATURE: [vmui](./README.md#vmui): retain specified time interval when executing a query from `Top Queries`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5097). -* FEATURE: [vmui](./README.md#vmui): improve repeated VMUI page load times by enabling caching of static js and css at web browser side according to [these recommendations](https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/). -* FEATURE: [vmui](./README.md#vmui): sort legend under the graph in descending order of median values. This should simplify graph analysis, since usually the most important lines have bigger values. -* FEATURE: [vmui](./README.md#vmui): reduce vertical space usage, so more information is visible on the screen without scrolling. -* FEATURE: [vmui](./README.md#vmui): show query execution duration in the header of query input field. This should help optimizing query performance. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add support for functions, labels, values in autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): retain specified time interval when executing a query from `Top Queries`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5097). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve repeated VMUI page load times by enabling caching of static js and css at web browser side according to [these recommendations](https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): sort legend under the graph in descending order of median values. This should simplify graph analysis, since usually the most important lines have bigger values. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): reduce vertical space usage, so more information is visible on the screen without scrolling. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show query execution duration in the header of query input field. This should help optimizing query performance. * FEATURE: support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options` HTTP response headers in the all VictoriaMetrics components. The values for headers can be specified via the following command-line flags: `-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`. -* FEATURE: [vmalert-tool](./vmalert-tool.md): add `unittest` command to run unittest for alerting and recording rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4789) for details. +* FEATURE: [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool.html): add `unittest` command to run unittest for alerting and recording rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4789) for details. * FEATURE: dashboards/vmalert: add new panel `Missed evaluations` for indicating alerting groups that miss their evaluations. -* FEATURE: all: track requests with wrong auth key and wrong basic auth at `vm_http_request_errors_total` [metric](./README.md#monitoring) with `reason="wrong_auth_key"` and `reason="wrong_basic_auth"`. See [this issue](https://github.com/victoriaMetrics/victoriaMetrics/issues/4590). Thanks to @venkatbvc for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5166). -* FEATURE: [vmauth](./vmauth.md): add ability to drop the specified number of `/`-delimited prefix parts from the request path before proxying the request to the matching backend. See [these docs](./vmauth.md#dropping-request-path-prefix). -* FEATURE: [vmauth](./vmauth.md): add ability to skip TLS verification and to specify TLS Root CA when connecting to backends. See [these docs](./vmauth.md#backend-tls-setup) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240). -* FEATURE: `vmstorage`: gradually close `vminsert` connections during 25 seconds at [graceful shutdown](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes). This should reduce data ingestion slowdown during rolling restarts. The duration for gradual closing of `vminsert` connections can be configured via `-storage.vminsertConnsShutdownDuration` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922) and [these docs](./Cluster-VictoriaMetrics.md#improving-re-routing-performance-during-restart) for details. +* FEATURE: all: track requests with wrong auth key and wrong basic auth at `vm_http_request_errors_total` [metric](https://docs.victoriametrics.com/#monitoring) with `reason="wrong_auth_key"` and `reason="wrong_basic_auth"`. See [this issue](https://github.com/victoriaMetrics/victoriaMetrics/issues/4590). Thanks to @venkatbvc for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5166). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to drop the specified number of `/`-delimited prefix parts from the request path before proxying the request to the matching backend. See [these docs](https://docs.victoriametrics.com/vmauth/#dropping-request-path-prefix). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to skip TLS verification and to specify TLS Root CA when connecting to backends. See [these docs](https://docs.victoriametrics.com/vmauth/#backend-tls-setup) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240). +* FEATURE: `vmstorage`: gradually close `vminsert` connections during 25 seconds at [graceful shutdown](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes). This should reduce data ingestion slowdown during rolling restarts. The duration for gradual closing of `vminsert` connections can be configured via `-storage.vminsertConnsShutdownDuration` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922) and [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart) for details. * FEATURE: `vmstorage`: add `-blockcache.missesBeforeCaching` command-line flag, which can be used for fine-tuning RAM usage for `indexdb/dataBlocks` cache when queries touching big number of time series are executed. * FEATURE: add `-loggerMaxArgLen` command-line flag for fine-tuning the maximum lengths of logged args. -* BUGFIX: [vmalert](./vmalert.md): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044). -* BUGFIX: [vmalert](./vmalert.md): fix vmalert web UI when running on 32-bit architectures machine. -* BUGFIX: [vmalert](./vmalert.md): do not send requests to configured remote systems when `-datasource.*`, `-remoteWrite.*`, `-remoteRead.*` or `-notifier.*` command-line flags refer files with invalid auth configs. Previously such requests were sent without properly set auth headers. Now the requests are sent only after the files are updated with valid auth configs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). -* BUGFIX: [vmalert](./vmalert.md): properly maintain alerts state in [replay mode](./vmalert.md#rules-backfilling) if alert's `for` param was bigger than replay request range (usually a couple of hours). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5186) for details. -* BUGFIX: [vmalert](./vmalert.md): increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded. -* BUGFIX: [vmalert](./vmalert.md): increment `vmalert_remotewrite_dropped_rows_total` metric if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix vmalert web UI when running on 32-bit architectures machine. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): do not send requests to configured remote systems when `-datasource.*`, `-remoteWrite.*`, `-remoteRead.*` or `-notifier.*` command-line flags refer files with invalid auth configs. Previously such requests were sent without properly set auth headers. Now the requests are sent only after the files are updated with valid auth configs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly maintain alerts state in [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling) if alert's `for` param was bigger than replay request range (usually a couple of hours). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5186) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): increment `vmalert_remotewrite_dropped_rows_total` metric if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls. * BUGFIX: `vmselect`: improve performance and memory usage during query processing on machines with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087). * BUGFIX: dashboards: fix vminsert/vmstorage/vmselect metrics filtering when dashboard is used to display data from many sub-clusters with unique job names. Before, only one specific job could have been accounted for component-specific panels, instead of all available jobs for the component. * BUGFIX: dashboards: respect `job` and `instance` filters for `alerts` annotation in cluster and single-node dashboards. @@ -144,245 +144,245 @@ Released at 2023-11-15 * BUGFIX: dashboards/vmalert: properly apply time series filter for panel `No data errors`. Before, the panel didn't respect `job` or `instance` filters. * BUGFIX: dashboards/vmalert: fix panel `Errors rate to Alertmanager` not showing any data due to wrong label filters. * BUGFIX: dashboards/cluster: fix description about `max` threshold for `Concurrent selects` panel. Before, it was mistakenly implying that `max` is equal to the double of available CPUs. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154) for details -* BUGFIX: [vmbackup](./vmbackup.md): fix error when creating an incremental backup with the `-origin` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5144) for details. -* BUGFIX: [vmagent](./vmagent.md): properly apply [relabeling](./vmagent.md#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). -* BUGFIX: [vmagent](./vmagent.md): properly discover Kubernetes targets via [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). Previously some targets and some labels could be skipped during service discovery because of the bug introduced in [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216) for more details. -* BUGFIX: [vmagent](./vmagent.md): fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5178). -* BUGFIX: [vmagent](./vmagent.md): do not scrape targets if the corresponding [scrape_configs](./sd_configs.md#scrape_configs) refer to files with invalid auth configs. Previously the targets were scraped without properly set auth headers in this case. Now targets are scraped only after the files are updated with valid auth configs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). -* BUGFIX: [vmagent](./vmagent.md): properly parse `ca`, `cert` and `key` options at `tls_config` section inside [http client settings](./sd_configs.md#http-api-client-options). Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. -* BUGFIX: [vmagent](./vmagent.md): properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207). -* BUGFIX: [vmagent](./vmagent.md): do not print redundant error logs when failed to scrape consul or nomad target. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5239). -* BUGFIX: [vmagent](./vmagent.md): generate proper link to the main page and to `favicon.ico` at http pages served by `vmagent` such as `/targets` or `/service-discovery` when `vmagent` sits behind an http proxy with custom http path prefixes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5306). -* BUGFIX: [vmagent](./vmagent.md): properly decode Snappy-encoded data blocks received via [VictoriaMetrics remote_write protocol](./vmagent.md#victoriametrics-remote-write-protocol). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5174). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. -* BUGFIX: [vmauth](./vmauth.md): show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236) for details. -* BUGFIX: [vmauth](./vmauth.md): properly proxy requests to HTTP/2.0 backends and properly pass `Host` header to backends. -* BUGFIX: [vmui](./README.md#vmui): fix the `Disable cache` toggle at `JSON` and `Table` views. Previously response caching was always enabled and couldn't be disabled at these views. -* BUGFIX: [vmui](./README.md#vmui): correctly display query errors on [Explore Prometheus Metrics](./README.md#metrics-explorer) page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5202) for details. -* BUGFIX: [vmui](./README.md#vmui): properly handle trailing slash in the server URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5203). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): correctly print error in logs when copying backup fails. Previously, error was displayed in metrics but was missing in logs. -* BUGFIX: fix panic, which could occur when [query tracing](./README.md#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154) for details +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix error when creating an incremental backup with the `-origin` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5144) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover Kubernetes targets via [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously some targets and some labels could be skipped during service discovery because of the bug introduced in [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216) for more details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5178). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not scrape targets if the corresponding [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) refer to files with invalid auth configs. Previously the targets were scraped without properly set auth headers in this case. Now targets are scraped only after the files are updated with valid auth configs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly parse `ca`, `cert` and `key` options at `tls_config` section inside [http client settings](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not print redundant error logs when failed to scrape consul or nomad target. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5239). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): generate proper link to the main page and to `favicon.ico` at http pages served by `vmagent` such as `/targets` or `/service-discovery` when `vmagent` sits behind an http proxy with custom http path prefixes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5306). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly decode Snappy-encoded data blocks received via [VictoriaMetrics remote_write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5174). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236) for details. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): properly proxy requests to HTTP/2.0 backends and properly pass `Host` header to backends. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the `Disable cache` toggle at `JSON` and `Table` views. Previously response caching was always enabled and couldn't be disabled at these views. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): correctly display query errors on [Explore Prometheus Metrics](https://docs.victoriametrics.com/#metrics-explorer) page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5202) for details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly handle trailing slash in the server URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5203). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): correctly print error in logs when copying backup fails. Previously, error was displayed in metrics but was missing in logs. +* BUGFIX: fix panic, which could occur when [query tracing](https://docs.victoriametrics.com/#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). ## [v1.94.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.94.0) Released at 2023-10-02 -* FEATURE: [MetricsQL](./MetricsQL.md): add support for numbers with underscore delimiters such as `1_234_567_890` and `1.234_567_890`. These numbers are easier to read than `1234567890` and `1.234567890`. -* FEATURE: [vmbackup](./vmbackup.md): add support for server-side copy of existing backups. See [these docs](./vmbackup.md#server-side-copy-of-the-existing-backup) for details. -* FEATURE: [vmui](./README.md#vmui): add the option to see the latest 25 queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4718). -* FEATURE: [vmagent](./vmagent.md): add ability to set `member num` label for all the metrics scraped by a particular `vmagent` instance in [a cluster of vmagents](./vmagent.md#scraping-big-number-of-targets) via `-promscrape.cluster.memberLabel` command-line flag. See [these docs](./vmagent.md#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247). -* FEATURE: [vmagent](./vmagent.md): do not log `unexpected EOF` when reading incoming metrics, since this error is expected and is handled during metrics' parsing. This reduces the amounts of noisy logs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4817). -* FEATURE: [vmagent](./vmagent.md): retry failed write request on the closed connection immediately, without waiting for backoff. This should improve data delivery speed and reduce amount of error logs emitted by vmagent when using idle connections. See related [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139). -* FEATURE: [vmagent](./vmagent.md): reduces load on Kubernetes control plane during initial service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4855) for details. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): reduce the maximum recovery time at `vmselect` and `vminsert` when some of `vmstorage` nodes become unavailable because of networking issues from 60 seconds to 3 seconds by default. The recovery time can be tuned at `vmselect` and `vminsert` nodes with `-vmstorageUserTimeout` command-line flag if needed. Thanks to @wjordan for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4423). -* FEATURE: [vmui](./README.md#vmui): add Prometheus data support to the "Explore cardinality" page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4320) for details. -* FEATURE: [vmui](./README.md#vmui): make the warning message more noticeable for text fields. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4848). -* FEATURE: [vmui](./README.md#vmui): add button for auto-formatting PromQL/MetricsQL queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4681). Thanks to @aramattamara for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4694). -* FEATURE: [vmui](./README.md#vmui): improve accessibility score to 100 according to [Google's Lighthouse](https://developer.chrome.com/docs/lighthouse/accessibility/) tests. -* FEATURE: [vmui](./README.md#vmui): organize `min`, `max`, `median` values on the chart legend and tooltips for better visibility. -* FEATURE: [vmui](./README.md#vmui): add explanation about [cardinality explorer](./README.md#cardinality-explorer) statistic inaccuracy in VictoriaMetrics cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3070). -* FEATURE: [vmui](./README.md#vmui): add storage of query history in `localStorage`. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5022). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add support for numbers with underscore delimiters such as `1_234_567_890` and `1.234_567_890`. These numbers are easier to read than `1234567890` and `1.234567890`. +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): add support for server-side copy of existing backups. See [these docs](https://docs.victoriametrics.com/vmbackup/#server-side-copy-of-the-existing-backup) for details. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the option to see the latest 25 queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4718). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to set `member num` label for all the metrics scraped by a particular `vmagent` instance in [a cluster of vmagents](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) via `-promscrape.cluster.memberLabel` command-line flag. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): do not log `unexpected EOF` when reading incoming metrics, since this error is expected and is handled during metrics' parsing. This reduces the amounts of noisy logs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4817). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): retry failed write request on the closed connection immediately, without waiting for backoff. This should improve data delivery speed and reduce amount of error logs emitted by vmagent when using idle connections. See related [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduces load on Kubernetes control plane during initial service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4855) for details. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): reduce the maximum recovery time at `vmselect` and `vminsert` when some of `vmstorage` nodes become unavailable because of networking issues from 60 seconds to 3 seconds by default. The recovery time can be tuned at `vmselect` and `vminsert` nodes with `-vmstorageUserTimeout` command-line flag if needed. Thanks to @wjordan for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4423). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add Prometheus data support to the "Explore cardinality" page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4320) for details. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): make the warning message more noticeable for text fields. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4848). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add button for auto-formatting PromQL/MetricsQL queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4681). Thanks to @aramattamara for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4694). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve accessibility score to 100 according to [Google's Lighthouse](https://developer.chrome.com/docs/lighthouse/accessibility/) tests. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): organize `min`, `max`, `median` values on the chart legend and tooltips for better visibility. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add explanation about [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) statistic inaccuracy in VictoriaMetrics cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3070). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add storage of query history in `localStorage`. See [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5022). * FEATURE: dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource at [dashboards/vm](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards/vm). -* FEATURE: [vmauth](./vmauth.md): added ability to set, override and clear request and response headers on a per-user and per-path basis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825) and [these docs](./vmauth.md#auth-config) for details. -* FEATURE: [vmauth](./vmauth.md): add ability to retry requests to the [remaining backends](./vmauth.md#load-balancing) if they return response status codes specified in the `retry_status_codes` list. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893). -* FEATURE: [vmauth](./vmauth.md): expose metrics `vmauth_config_last_reload_*` for tracking the state of config reloads, similarly to vmagent/vmalert components. -* FEATURE: [vmauth](./vmauth.md): do not print logs like `SIGHUP received...` once per configured `-configCheckInterval` cmd-line flag. This log will be printed only if config reload was invoked manually. -* FEATURE: [vmalert](./vmalert.md): add `eval_offset` attribute for [Groups](./vmalert.md#groups). If specified, Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]. The setting might be useful for cron-like rules which must be evaluated at specific moments of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409) for details. -* FEATURE: [vmalert](./vmalert.md): validate [MetricsQL](./MetricsQL.md) function names in alerting and recording rules when `vmalert` runs with `-dryRun` command-line flag. Previously it was allowed to use unknown (aka invalid) MetricsQL function names there. For example, `foo()` was counted as a valid query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4933). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): added ability to set, override and clear request and response headers on a per-user and per-path basis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825) and [these docs](https://docs.victoriametrics.com/vmauth/#auth-config) for details. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to retry requests to the [remaining backends](https://docs.victoriametrics.com/vmauth/#load-balancing) if they return response status codes specified in the `retry_status_codes` list. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): expose metrics `vmauth_config_last_reload_*` for tracking the state of config reloads, similarly to vmagent/vmalert components. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): do not print logs like `SIGHUP received...` once per configured `-configCheckInterval` cmd-line flag. This log will be printed only if config reload was invoked manually. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `eval_offset` attribute for [Groups](https://docs.victoriametrics.com/vmalert/#groups). If specified, Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]. The setting might be useful for cron-like rules which must be evaluated at specific moments of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): validate [MetricsQL](https://docs.victoriametrics.com/metricsql/) function names in alerting and recording rules when `vmalert` runs with `-dryRun` command-line flag. Previously it was allowed to use unknown (aka invalid) MetricsQL function names there. For example, `foo()` was counted as a valid query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4933). * FEATURE: limit the length of string params in log messages to 500 chars. Longer string params are replaced with the `first_250_chars..last_250_chars`. This prevents from too long log lines, which can be emitted by VictoriaMetrics components. * FEATURE: [docker compose environment](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker): add `vmauth` component to cluster's docker-compose example for balancing load among multiple `vmselect` components. -* FEATURE: [MetricsQL](./MetricsQL.md): make sure that `q2` series are returned after `q1` series in the results of `q1 or q2` query, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763). -* FEATURE: [MetricsQL](./MetricsQL.md): return empty result from [`bitmap_and(a, b)`](./MetricsQL.md#bitmap_and), [`bitmap_or(a, b)`](./MetricsQL.md#bitmap_or) and [`bitmap_xor(a, b)`](./MetricsQL.md#bitmap_xor) if `a` or `b` have no value at the particular timestamp. Previously `0` was returned in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): make sure that `q2` series are returned after `q1` series in the results of `q1 or q2` query, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): return empty result from [`bitmap_and(a, b)`](https://docs.victoriametrics.com/metricsql/#bitmap_and), [`bitmap_or(a, b)`](https://docs.victoriametrics.com/metricsql/#bitmap_or) and [`bitmap_xor(a, b)`](https://docs.victoriametrics.com/metricsql/#bitmap_xor) if `a` or `b` have no value at the particular timestamp. Previously `0` was returned in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996). * FEATURE: stop exposing `vm_merge_need_free_disk_space` metric, since it has been appeared that it confuses users while doesn't bring any useful information. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686#issuecomment-1733844128). * BUGFIX: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): fix display of ingested rows rate for `Samples ingested/s` and `Samples rate` panels for vmagent's dashboard. Previously, not all ingested protocols were accounted in these panels. An extra panel `Rows rate` was added to `Ingestion` section to display the split for rows ingested rate by protocol. -* BUGFIX: [vmui](./README.md#vmui): fix the bug causing render looping when switching to heatmap. -* BUGFIX: [VictoriaMetrics enterprise](./enterprise.md) validate `-dedup.minScrapeInterval` value and `-downsampling.period` intervals are multiples of each other. See [these docs](./README.md#downsampling). -* BUGFIX: [vmbackup](./vmbackup.md): properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during [incremental backups](./vmbackup.md#incremental-backups). Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). -* BUGFIX: [vmagent](./vmagent.md): suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5). See [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5048). -* BUGFIX: [vmagent](./vmagent.md): remove concurrency limit during parsing of scraped metrics, which was mistakenly applied to it. With this change cmd-line flag `-maxConcurrentInserts` won't have effect on scraping anymore. -* BUGFIX: [MetricsQL](./MetricsQL.md): allow passing [median_over_time](./MetricsQL.md#median_over_time) to [aggr_over_time](./MetricsQL.md#aggr_over_time). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): fix ingestion via [multitenant url](./Cluster-VictoriaMetrics.md#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2). -* BUGFIX: [vmagent](./vmagent.md): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the bug causing render looping when switching to heatmap. +* BUGFIX: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) validate `-dedup.minScrapeInterval` value and `-downsampling.period` intervals are multiples of each other. See [these docs](https://docs.victoriametrics.com/#downsampling). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during [incremental backups](https://docs.victoriametrics.com/vmbackup/#incremental-backups). Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5). See [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5048). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): remove concurrency limit during parsing of scraped metrics, which was mistakenly applied to it. With this change cmd-line flag `-maxConcurrentInserts` won't have effect on scraping anymore. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow passing [median_over_time](https://docs.victoriametrics.com/metricsql/#median_over_time) to [aggr_over_time](https://docs.victoriametrics.com/metricsql/#aggr_over_time). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): fix ingestion via [multitenant url](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. ## [v1.93.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.9) Released at 2023-12-10 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See [alpine 3.19.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html). * SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See [the list of issues addressed in Go1.21.5](https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): prevent from `FATAL: cannot flush metainfo` panic when [`-remoteWrite.multitenantURL`](./vmagent.md#multitenancy) command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357). -* BUGFIX: [vmagent](./vmagent.md): properly decode zstd-encoded data blocks received via [VictoriaMetrics remote_write protocol](./vmagent.md#victoriametrics-remote-write-protocol). See [this issue comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301#issuecomment-1815871992). -* BUGFIX: [vmagent](./vmagent.md): properly add new labels at `output_relabel_configs` during [stream aggregation](./stream-aggregation.md). Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing [detailed report for this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5402). -* BUGFIX: [vmalert](./vmalert.md): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. -* BUGFIX: properly escape `<` character in responses returned via [`/federate`](./README.md#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent from `FATAL: cannot flush metainfo` panic when [`-remoteWrite.multitenantURL`](https://docs.victoriametrics.com/vmagent/#multitenancy) command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly decode zstd-encoded data blocks received via [VictoriaMetrics remote_write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). See [this issue comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301#issuecomment-1815871992). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly add new labels at `output_relabel_configs` during [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/). Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing [detailed report for this bug](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5402). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. +* BUGFIX: properly escape `<` character in responses returned via [`/federate`](https://docs.victoriametrics.com/#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). ## [v1.93.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.8) Released at 2023-11-15 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.3 to Go1.21.4. See [the list of issues addressed in Go1.21.4](https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): properly apply [relabeling](./vmagent.md#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). -* BUGFIX: [vmagent](./vmagent.md): properly decode Snappy-encoded data blocks received via [VictoriaMetrics remote_write protocol](./vmagent.md#victoriametrics-remote-write-protocol). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301). -* BUGFIX: fix panic, which could occur when [query tracing](./README.md#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly decode Snappy-encoded data blocks received via [VictoriaMetrics remote_write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301). +* BUGFIX: fix panic, which could occur when [query tracing](https://docs.victoriametrics.com/#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). ## [v1.93.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.7) Released at 2023-11-02 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -* BUGFIX: [vmagent](./vmagent.md): properly discover Kubernetes targets via [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). Previously some targets and some labels could be skipped during service discovery because of the bug introduced in [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216) for more details. -* BUGFIX: [vmagent](./vmagent.md): properly parse `ca`, `cert` and `key` options at `tls_config` section inside [http client settings](./sd_configs.md#http-api-client-options). Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. -* BUGFIX: [vmagent](./vmagent.md): properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207). -* BUGFIX: [vmagent](./vmagent.md): do not print redundant error logs when failed to scrape consul or nomad target. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5239). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5174). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. -* BUGFIX: [vmauth](./vmauth.md): show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly discover Kubernetes targets via [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously some targets and some labels could be skipped during service discovery because of the bug introduced in [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216) for more details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly parse `ca`, `cert` and `key` options at `tls_config` section inside [http client settings](https://docs.victoriametrics.com/sd_configs/#http-api-client-options). Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not print redundant error logs when failed to scrape consul or nomad target. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5239). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069) and [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5174). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236) for details. ## [v1.93.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.6) Released at 2023-10-16 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See [the list of issues addressed in Go1.21.2](https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved) and [the list of issues addressed in Go1.21.3](https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved). -* BUGFIX: [vmalert](./vmalert.md): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044). * BUGFIX: `vmselect`: improve performance and memory usage during query processing on machines with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087) for details. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154) for details -* BUGFIX: [vmagent](./vmagent.md): fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5178). -* BUGFIX: [vmbackup](./vmbackup.md): properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during [incremental backups](./vmbackup.md#incremental-backups). Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). -* BUGFIX: [vmagent](./vmagent.md): suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5). See [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5048). -* BUGFIX: [MetricsQL](./MetricsQL.md): allow passing [median_over_time](./MetricsQL.md#median_over_time) to [aggr_over_time](./MetricsQL.md#aggr_over_time). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): fix ingestion via [multitenant url](./Cluster-VictoriaMetrics.md#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2). -* BUGFIX: [vmagent](./vmagent.md): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154) for details +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5178). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during [incremental backups](https://docs.victoriametrics.com/vmbackup/#incremental-backups). Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5). See [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5048). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow passing [median_over_time](https://docs.victoriametrics.com/metricsql/#median_over_time) to [aggr_over_time](https://docs.victoriametrics.com/metricsql/#aggr_over_time). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): fix ingestion via [multitenant url](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. ## [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5) Released at 2023-09-19 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -* BUGFIX: [storage](./Single-Server-VictoriaMetrics.md): prevent from livelock when [forced merge](./README.md#forced-merge) is called under high data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4987). -* BUGFIX: [Graphite Render API](./README.md#graphite-render-api-usage): correctly return `null` instead of `Inf` in JSON query responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3783). -* BUGFIX: [vmbackup](./vmbackup.md): properly copy `parts.json` files inside `<-storageDataPath>/{data,indexdb}` folders during [incremental backups](./vmbackup.md#incremental-backups). Previously the new `parts.json` could be skipped during incremental backups, which could lead to inability to restore from the backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). This issue has been introduced in [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0). -* BUGFIX: [vmagent](./vmagent.md): properly close connections to Kubernetes API server after the change in `selectors` or `namespaces` sections of [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). Previously `vmagent` could continue polling Kubernetes API server with the old `selectors` or `namespaces` configs additionally to polling new configs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). -* BUGFIX: [vmauth](./vmauth.md): prevent configuration reloading if there were no changes in config. This improves memory usage when `-configCheckInterval` cmd-line flag is configured and config has extensive list of regexp expressions requiring additional memory on parsing. +* BUGFIX: [storage](https://docs.victoriametrics.com/single-server-victoriametrics/): prevent from livelock when [forced merge](https://docs.victoriametrics.com/#forced-merge) is called under high data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4987). +* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): correctly return `null` instead of `Inf` in JSON query responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3783). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly copy `parts.json` files inside `<-storageDataPath>/{data,indexdb}` folders during [incremental backups](https://docs.victoriametrics.com/vmbackup/#incremental-backups). Previously the new `parts.json` could be skipped during incremental backups, which could lead to inability to restore from the backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). This issue has been introduced in [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly close connections to Kubernetes API server after the change in `selectors` or `namespaces` sections of [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). Previously `vmagent` could continue polling Kubernetes API server with the old `selectors` or `namespaces` configs additionally to polling new configs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): prevent configuration reloading if there were no changes in config. This improves memory usage when `-configCheckInterval` cmd-line flag is configured and config has extensive list of regexp expressions requiring additional memory on parsing. ## [v1.93.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.4) Released at 2023-09-10 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * SECURITY: upgrade Go builder from Go1.21.0 to Go1.21.1. See [the list of issues addressed in Go1.20.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved). -* BUGFIX: [vminsert enterprise](./enterprise.md): properly parse `/insert/multitenant/*` urls, which have been broken since [v1.93.2](#v1932). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4947). +* BUGFIX: [vminsert enterprise](https://docs.victoriametrics.com/enterprise/): properly parse `/insert/multitenant/*` urls, which have been broken since [v1.93.2](#v1932). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4947). * BUGFIX: properly build production armv5 binaries for `GOARCH=arm`. This has been broken after the upgrading of Go builder to Go1.21.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965). -* BUGFIX: [vmselect](./Cluster-VictoriaMetrics.md): return `503 Service Unavailable` status code when [partial responses](./Cluster-VictoriaMetrics.md#cluster-availability) are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entity` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. -* BUGFIX: [vmalert](./vmalert.md): fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/eccecdf177115297fa1dc4d42d38e23de9a9f2cb) starting from [v1.91.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908). -* BUGFIX: [vmagent](./vmagent.md): fix possible corruption of labels in the collected samples if `-remoteWrite.label` is set together with multiple `-remoteWrite.url` options. The bug has been introduced in [v1.93.1](#v1931). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4972). +* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): return `503 Service Unavailable` status code when [partial responses](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability) are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entity` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/eccecdf177115297fa1dc4d42d38e23de9a9f2cb) starting from [v1.91.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible corruption of labels in the collected samples if `-remoteWrite.label` is set together with multiple `-remoteWrite.url` options. The bug has been introduced in [v1.93.1](#v1931). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4972). ## [v1.93.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.3) Released at 2023-09-02 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): properly close broken vmstorage connection during [read-only state](./Cluster-VictoriaMetrics.md#readonly-mode) checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4932). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): properly close broken vmstorage connection during [read-only state](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode) checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4932). ## [v1.93.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.2) Released at 2023-09-01 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -* BUGFIX: [build](./README.md): fix Docker builds for old Docker releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907). -* BUGFIX: [vmagent](./vmagent.md): consistently set `User-Agent` header to `vm_promscrape` during scraping with enabled or disabled [stream parsing mode](./vmagent.md#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4884). -* BUGFIX: [vmagent](./vmagent.md): consistently set timeout for scraping with enabled or disabled [stream parsing mode](./vmagent.md#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4847). -* BUGFIX: [vmalert](./vmalert.md): correctly re-use HTTP request object on `EOF` retries when querying the configured datasource. Previously, there was a small chance that query retry wouldn't succeed. -* BUGFIX: [vmselect](./Cluster-VictoriaMetrics.md): correctly handle requests with `/select/multitenant` prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): properly check for [read-only state](./Cluster-VictoriaMetrics.md#readonly-mode) at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). +* BUGFIX: [build](https://docs.victoriametrics.com/): fix Docker builds for old Docker releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): consistently set `User-Agent` header to `vm_promscrape` during scraping with enabled or disabled [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4884). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): consistently set timeout for scraping with enabled or disabled [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4847). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): correctly re-use HTTP request object on `EOF` retries when querying the configured datasource. Previously, there was a small chance that query retry wouldn't succeed. +* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): correctly handle requests with `/select/multitenant` prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): properly check for [read-only state](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode) at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). ## [v1.93.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.1) Released at 2023-08-23 -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** * BUGFIX: prevent from possible data loss during `indexdb` rotation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details. * BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form `-boolFlagName value`, since this leads to silent incomplete flags' parsing. This form should be replaced with `-boolFlagName=value`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4845). -* BUGFIX: [vmagent](./vmagent.md): properly set labels from `-remoteWrite.label` command-line flag just before sending samples to the configured `-remoteWrite.url` according to [these docs](./vmagent.md#adding-labels-to-metrics). Previously these labels were incorrectly set before [the relabeling](./vmagent.md#relabeling) configured via `-remoteWrite.urlRelabelConfigs` and [the stream aggregation](./stream-aggregation.md) configured via `-remoteWrite.streamAggr.config`, so these labels could be lost or incorrectly transformed before sending the samples to remote storage. The fix allows using `-remoteWrite.label` for identifying `vmagent` instances in [cluster mode](./vmagent.md#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247) and [these docs](./stream-aggregation.md#cluster-mode) for more details. -* BUGFIX: remove `DEBUG` logging when parsing `if` filters inside [relabeling rules](./vmagent.md#relabeling-enhancements) and when parsing `match` filters inside [stream aggregation rules](./stream-aggregation.md). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set labels from `-remoteWrite.label` command-line flag just before sending samples to the configured `-remoteWrite.url` according to [these docs](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics). Previously these labels were incorrectly set before [the relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) configured via `-remoteWrite.urlRelabelConfigs` and [the stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configured via `-remoteWrite.streamAggr.config`, so these labels could be lost or incorrectly transformed before sending the samples to remote storage. The fix allows using `-remoteWrite.label` for identifying `vmagent` instances in [cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247) and [these docs](https://docs.victoriametrics.com/stream-aggregation/#cluster-mode) for more details. +* BUGFIX: remove `DEBUG` logging when parsing `if` filters inside [relabeling rules](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements) and when parsing `match` filters inside [stream aggregation rules](https://docs.victoriametrics.com/stream-aggregation/). * BUGFIX: properly replace `:` chars in label names with `_` when `-usePromCompatibleNaming` command-line flag is passed to `vmagent`, `vminsert` or single-node VictoriaMetrics. This addresses [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113#issuecomment-1275077071). -* BUGFIX: [vmbackup](./vmbackup.md): correctly check if specified `-dst` belongs to specified `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837). -* BUGFIX: [vmctl](./vmctl.md): don't interrupt the migration process if no metrics were found for a specific tenant. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4796). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): correctly check if specified `-dst` belongs to specified `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): don't interrupt the migration process if no metrics were found for a specific tenant. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4796). ## [v1.93.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.0) Released at 2023-08-12 -**It is recommended upgrading to [VictoriaMetrics v1.93.1](./CHANGELOG.md#v1931) because v1.93.0 contains a bug, which can lead to data loss because of incorrect `indexdb` rotation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details.** +**It is recommended upgrading to [VictoriaMetrics v1.93.1](https://docs.victoriametrics.com/changelog/#v1931) because v1.93.0 contains a bug, which can lead to data loss because of incorrect `indexdb` rotation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details.** -**v1.93.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.93.x line will be supported for at least 12 months since [v1.93.0](./CHANGELOG.md#v1930) release** +**v1.93.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/changelog/#v1930) release** -**Update note**: starting from this release, [vmagent](./vmagent.md) ignores timestamps provided by scrape targets by default - it associates scraped metrics with local timestamps instead. Set `honor_timestamps: true` in [scrape configs](./sd_configs.md#scrape_configs) if timestamps provided by scrape targets must be used instead. This change helps removing gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) such as `container_memory_usage_bytes`. This also improves data compression and query performance over metrics collected from `cadvisor`. See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697). +**Update note**: starting from this release, [vmagent](https://docs.victoriametrics.com/vmagent/) ignores timestamps provided by scrape targets by default - it associates scraped metrics with local timestamps instead. Set `honor_timestamps: true` in [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) if timestamps provided by scrape targets must be used instead. This change helps removing gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) such as `container_memory_usage_bytes`. This also improves data compression and query performance over metrics collected from `cadvisor`. See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697). * SECURITY: upgrade Go builder from Go1.20.6 to Go1.21.0 in order to fix [this issue](https://github.com/golang/go/issues/61460). * SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See [alpine 3.18.3 release notes](https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html). -* FEATURE: [MetricsQL](./MetricsQL.md): add `share_eq_over_time(m[d], eq)` function for calculating the share (in the range `[0...1]`) of raw samples on the given lookbehind window `d`, which are equal to `eq`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4441). Thanks to @Damon07 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4725). -* FEATURE: [vmauth](./vmauth.md): allow configuring deadline for a backend to be excluded from the rotation on errors via `-failTimeout` cmd-line flag. This feature could be useful when it is expected for backends to be not available for significant periods of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4415) for details. Thanks to @SunKyu for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4416). -* FEATURE: [vmalert](./vmalert.md): remove deprecated in [v1.61.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.0) `-rule.configCheckInterval` command-line flag. Use `-configCheckInterval` command-line flag instead. -* FEATURE: [vmalert](./vmalert.md): remove support of deprecated web links of `/api/v1///status` form in favour of `/api/v1/alerts?group_id=<>&alert_id=<>` links. Links of `/api/v1///status` form were deprecated in v1.79.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825) for details. -* FEATURE: [vmctl](./vmctl.md): allow disabling binary export API protocol via `-vm-native-disable-binary-protocol` cmd-line flag when [migrating data from VictoriaMetrics](./vmctl.md#migrating-data-from-victoriametrics). Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need [to be configured](./README.md#deduplication) at `-vm-native-src-addr` side and `-vm-native-disable-binary-protocol` should be set on vmctl side. -* FEATURE: [vmctl](./vmctl.md): add support of `week` step for [time-based chunking migration](./vmctl.md#using-time-based-chunking-of-migration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738). -* FEATURE: [vmctl](./vmctl.md): allow specifying custom full url at `--remote-read-src-addr` command-line flag if `--remote-read-disable-path-append` command-line flag is set. This allows importing data from urls, which do not end with `/api/v1/read`. For example, from Promscale. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655). -* FEATURE: [vmui](./README.md#vmui): add warning in query field of vmui for partial data responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4721). -* FEATURE: [vmui](./README.md#vmui): allow displaying the full error message on click for trimmed error messages in vmui. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4719). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `share_eq_over_time(m[d], eq)` function for calculating the share (in the range `[0...1]`) of raw samples on the given lookbehind window `d`, which are equal to `eq`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4441). Thanks to @Damon07 for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4725). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow configuring deadline for a backend to be excluded from the rotation on errors via `-failTimeout` cmd-line flag. This feature could be useful when it is expected for backends to be not available for significant periods of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4415) for details. Thanks to @SunKyu for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4416). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): remove deprecated in [v1.61.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.0) `-rule.configCheckInterval` command-line flag. Use `-configCheckInterval` command-line flag instead. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): remove support of deprecated web links of `/api/v1///status` form in favour of `/api/v1/alerts?group_id=<>&alert_id=<>` links. Links of `/api/v1///status` form were deprecated in v1.79.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825) for details. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): allow disabling binary export API protocol via `-vm-native-disable-binary-protocol` cmd-line flag when [migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need [to be configured](https://docs.victoriametrics.com/#deduplication) at `-vm-native-src-addr` side and `-vm-native-disable-binary-protocol` should be set on vmctl side. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add support of `week` step for [time-based chunking migration](https://docs.victoriametrics.com/vmctl/#using-time-based-chunking-of-migration). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): allow specifying custom full url at `--remote-read-src-addr` command-line flag if `--remote-read-disable-path-append` command-line flag is set. This allows importing data from urls, which do not end with `/api/v1/read`. For example, from Promscale. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add warning in query field of vmui for partial data responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4721). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): allow displaying the full error message on click for trimmed error messages in vmui. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4719). * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add `Concurrent inserts` panel to vmagent's dashboard. The new panel supposed to show whether the number of concurrent inserts processed by vmagent isn't reaching the limit. * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add panels for absolute Mem and CPU usage by vmalert. See related issue [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4627). * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): correctly calculate `Bytes per point` value for single-server and cluster VM dashboards. Before, the calculation mistakenly accounted for the number of entries in indexdb in denominator, which could have shown lower values than expected. * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): `ConcurrentFlushesHitTheLimit` alerting rule was moved from [single-server](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) and [cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-cluster.yml) alerts to the [list of "health" alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml) as it could be related to many VictoriaMetrics components. -* BUGFIX: [storage](./Single-Server-VictoriaMetrics.md): properly set next retention time for indexDB. Previously it may enter into endless retention loop. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details. -* BUGFIX: [vmagent](./vmagent.md): return human readable error if opentelemetry has json encoding. Follow-up after [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570). -* BUGFIX: [vmagent](./vmagent.md): properly validate scheme for `proxy_url` field at the scrape config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4811) for details. -* BUGFIX: [vmagent](./vmagent.md): properly apply `if` filters during [relabeling](./vmagent.md#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). -* BUGFIX: [vmagent](./vmagent.md): use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at [scrape_config](./sd_configs.md#scrape_configs). This fixes gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) or similar exporters, which export metrics with invalid timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799) for details. The issue has been introduced in [v1.68.0](./CHANGELOG_2021.md#v1680). -* BUGFIX: [vmagent](./vmagent.md): fixes runtime panic at OpenTelemetry parser. Opentelemetry format allows histograms without `sum` fields. Such histogram converted as counter with `_count` suffix. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4814). -* BUGFIX: [vmagent](./vmagent.md): keep unmatched series at [stream aggregation](./stream-aggregation.md) when `-remoteWrite.streamAggr.dropInput` is set to `false` to match intended behaviour introduced at [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4804). -* BUGFIX: [vmalert](./vmalert.md): properly set `vmalert_config_last_reload_successful` value on configuration updates or rollbacks. The bug was introduced in [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0) in [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4543). -* BUGFIX: [vmalert](./vmalert.md): fix `vmalert_remotewrite_send_duration_seconds_total` value, before it didn't count in the real time spending on remote write requests. See [this pr](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4801) for details. -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix panic when creating a backup to a local filesystem on Windows. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704). -* BUGFIX: [vmui](./README.md#vmui): properly handle client address with `X-Forwarded-For` part at the [Active queries](./README.md#active-queries) page. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676#issuecomment-1663203424). -* BUGFIX: [MetricsQL](./MetricsQL.md): prevent from panic when the lookbehind window in square brackets of [rollup function](./MetricsQL.md#rollup-functions) is parsed into negative value. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4795). +* BUGFIX: [storage](https://docs.victoriametrics.com/single-server-victoriametrics/): properly set next retention time for indexDB. Previously it may enter into endless retention loop. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): return human readable error if opentelemetry has json encoding. Follow-up after [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly validate scheme for `proxy_url` field at the scrape config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4811) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply `if` filters during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). This fixes gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) or similar exporters, which export metrics with invalid timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799) for details. The issue has been introduced in [v1.68.0](https://docs.victoriametrics.com/changelog_2021/#v1680). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fixes runtime panic at OpenTelemetry parser. Opentelemetry format allows histograms without `sum` fields. Such histogram converted as counter with `_count` suffix. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4814). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): keep unmatched series at [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) when `-remoteWrite.streamAggr.dropInput` is set to `false` to match intended behaviour introduced at [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4804). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly set `vmalert_config_last_reload_successful` value on configuration updates or rollbacks. The bug was introduced in [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0) in [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4543). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix `vmalert_remotewrite_send_duration_seconds_total` value, before it didn't count in the real time spending on remote write requests. See [this pr](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4801) for details. +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix panic when creating a backup to a local filesystem on Windows. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly handle client address with `X-Forwarded-For` part at the [Active queries](https://docs.victoriametrics.com/#active-queries) page. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676#issuecomment-1663203424). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): prevent from panic when the lookbehind window in square brackets of [rollup function](https://docs.victoriametrics.com/metricsql/#rollup-functions) is parsed into negative value. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4795). ## [v1.92.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.1) Released at 2023-07-28 -* BUGFIX: [vmalert](./vmalert.md): revert unit test feature for alerting and recording rules introduced in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4596). See the following [change](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4734). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): revert unit test feature for alerting and recording rules introduced in [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4596). See the following [change](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4734). ## [v1.92.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.92.0) @@ -391,11 +391,11 @@ Released at 2023-07-27 **Update note: this release contains backwards-incompatible change to indexdb, so rolling back to the previous versions of VictoriaMetrics may result in partial data loss of entries in indexdb.** -**Update note**: starting from this release, [stream aggregation](./stream-aggregation.md) writes +**Update note**: starting from this release, [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) writes the following samples to the configured remote storage by default: - aggregated samples; -- the original input samples, which match zero `match` options from the provided [config](./stream-aggregation.md#stream-aggregation-config). +- the original input samples, which match zero `match` options from the provided [config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). Previously only aggregated samples were written to the storage by default. The previous behavior can be restored in the following ways: @@ -408,59 +408,59 @@ The previous behavior can be restored in the following ways: * SECURITY: upgrade base docker image (alpine) from 3.18.0 to 3.18.2. See [alpine 3.18.2 release notes](https://alpinelinux.org/posts/Alpine-3.15.9-3.16.6-3.17.4-3.18.2-released.html). * SECURITY: upgrade Go builder from Go1.20.5 to Go1.20.6. See [the list of issues addressed in Go1.20.6](https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved). -* FEATURE: reduce memory usage by up to 5x for setups with [high churn rate](./FAQ.md#what-is-high-churn-rate) and long [retention](./README.md#retention). See [the description for this change](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7094fa38bc207c7bd7330ea8a834310a310ce5e3) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563) for details. -* FEATURE: reduce spikes in CPU and disk IO usage during `indexdb` rotation (aka inverted index), which is performed once per [`-retentionPeriod`](./README.md#retention). The new algorithm gradually pre-populates newly created `indexdb` during the last hour before the rotation. The number of pre-populated series in the newly created `indexdb` can be [monitored](./README.md#monitoring) via `vm_timeseries_precreated_total` metric. This should resolve [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401). -* FEATURE: [MetricsQL](./MetricsQL.md): allow selecting time series matching at least one of multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}` selects series with either `{env="prod",job="a"}` or `{env="dev",job="b"}` labels. This functionality allows passing the selected series to [rollup functions](./MetricsQL.md#rollup-functions) without the need to use [subqueries](./MetricsQL.md#subqueries). See [these docs](./keyConcepts.md#filtering-by-multiple-or-filters). -* FEATURE: [MetricsQL](./MetricsQL.md): add ability to preserve metric names for binary operation results via `keep_metric_names` modifier. For example, `({__name__=~"foo|bar"} / 10) keep_metric_names` leaves `foo` and `bar` metric names in division results. See [these docs](./MetricsQL.md#keep_metric_names). This helps to address issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3710). -* FEATURE: [MetricsQL](./MetricsQL.md): add ability to copy all the labels from `one` side of [many-to-one operations](https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches) by specifying `*` inside `group_left()` or `group_right()`. Also allow adding a prefix for copied label names via `group_left(*) prefix "..."` syntax. For example, the following query copies Kubernetes namespace labels to `kube_pod_info` series and adds `ns_` prefix for the copied label names: `kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels`. The labels from `on()` list aren't prefixed. This feature resolves [this](https://stackoverflow.com/questions/76661818/how-to-add-namespace-labels-to-pod-labels-in-prometheus) and [that](https://stackoverflow.com/questions/76653997/how-can-i-make-a-new-copy-of-kube-namespace-labels-metric-with-a-different-name) questions at StackOverflow. -* FEATURE: [MetricsQL](./MetricsQL.md): add ability to specify durations via [`WITH` templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/expand-with-exprs). Examples: +* FEATURE: reduce memory usage by up to 5x for setups with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) and long [retention](https://docs.victoriametrics.com/#retention). See [the description for this change](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7094fa38bc207c7bd7330ea8a834310a310ce5e3) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563) for details. +* FEATURE: reduce spikes in CPU and disk IO usage during `indexdb` rotation (aka inverted index), which is performed once per [`-retentionPeriod`](https://docs.victoriametrics.com/#retention). The new algorithm gradually pre-populates newly created `indexdb` during the last hour before the rotation. The number of pre-populated series in the newly created `indexdb` can be [monitored](https://docs.victoriametrics.com/#monitoring) via `vm_timeseries_precreated_total` metric. This should resolve [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow selecting time series matching at least one of multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}` selects series with either `{env="prod",job="a"}` or `{env="dev",job="b"}` labels. This functionality allows passing the selected series to [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) without the need to use [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries). See [these docs](https://docs.victoriametrics.com/keyconcepts/#filtering-by-multiple-or-filters). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add ability to preserve metric names for binary operation results via `keep_metric_names` modifier. For example, `({__name__=~"foo|bar"} / 10) keep_metric_names` leaves `foo` and `bar` metric names in division results. See [these docs](https://docs.victoriametrics.com/metricsql/#keep_metric_names). This helps to address issues like [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3710). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add ability to copy all the labels from `one` side of [many-to-one operations](https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches) by specifying `*` inside `group_left()` or `group_right()`. Also allow adding a prefix for copied label names via `group_left(*) prefix "..."` syntax. For example, the following query copies Kubernetes namespace labels to `kube_pod_info` series and adds `ns_` prefix for the copied label names: `kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels`. The labels from `on()` list aren't prefixed. This feature resolves [this](https://stackoverflow.com/questions/76661818/how-to-add-namespace-labels-to-pod-labels-in-prometheus) and [that](https://stackoverflow.com/questions/76653997/how-can-i-make-a-new-copy-of-kube-namespace-labels-metric-with-a-different-name) questions at StackOverflow. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add ability to specify durations via [`WITH` templates](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/expand-with-exprs). Examples: - `WITH (w = 5m) m[w]` is automatically transformed to `m[5m]` - `WITH (f(window, step, off) = m[window:step] offset off) f(5m, 10s, 1h)` is automatically transformed to `m[5m:10s] offset 1h` Thanks to @lujiajing1126 for the initial idea and [implementation](https://github.com/VictoriaMetrics/metricsql/pull/13). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4025). -* FEATURE: [vmui](./README.md#vmui): added a new page with the list of currently running queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4598) and [these docs](./README.md#active-queries). -* FEATURE: [vmagent](./vmagent.md): add support for data ingestion via [OpenTelemetry protocol](https://opentelemetry.io/docs/reference/specification/metrics/). See [these docs](./Single-Server-VictoriaMetrics.md#sending-data-via-opentelemetry), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2424) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570). -* FEATURE: [vmagent](./vmagent.md): allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable [stream aggregation](./stream-aggregation.md), when samples for the same time series must be aggregated by the same `vmagent` instance at the second level. See [these docs](./vmagent.md#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637) for details. -* FEATURE: [vmagent](./vmagent.md): allow configuring staleness interval in [stream aggregation](./stream-aggregation.md) config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667) for details. -* FEATURE: [vmagent](./vmagent.md): allow specifying a list of [series selectors](./keyConcepts.md#filtering) inside `if` option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See [these docs](./vmagent.md#relabeling-enhancements). -* FEATURE: [stream aggregation](./stream-aggregation.md): allow specifying a list of [series selectors](./keyConcepts.md#filtering) inside `match` option of [stream aggregation configs](./stream-aggregation.md#stream-aggregation-config). The input sample is aggregated when at least a single series selector matches. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635). -* FEATURE: [stream aggregation](./stream-aggregation.md): preserve input samples, which match zero `match` options from the [configured aggregations](./stream-aggregation.md#stream-aggregation-config). Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics or by passing `-remoteWrite.streamAggr.dropInput` command-line flag to `vmagent`. -* FEATURE: [vmctl](./vmctl.md): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081). -* FEATURE: [vmctl](./vmctl.md): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442). -* FEATURE: [vmctl](./vmctl.md): update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4402). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): added a new page with the list of currently running queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4598) and [these docs](https://docs.victoriametrics.com/#active-queries). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for data ingestion via [OpenTelemetry protocol](https://opentelemetry.io/docs/reference/specification/metrics/). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#sending-data-via-opentelemetry), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2424) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/), when samples for the same time series must be aggregated by the same `vmagent` instance at the second level. See [these docs](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow configuring staleness interval in [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667) for details. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying a list of [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) inside `if` option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow specifying a list of [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) inside `match` option of [stream aggregation configs](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). The input sample is aggregated when at least a single series selector matches. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): preserve input samples, which match zero `match` options from the [configured aggregations](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics or by passing `-remoteWrite.streamAggr.dropInput` command-line flag to `vmagent`. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4402). * FEATURE: vmstorage: suppress "broken pipe" and "connection reset by peer" errors for search queries on vmstorage side. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418/commits/a6a7795b9e1f210d614a2c5f9a3016b97ded4792) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4498/commits/830dac177f0f09032165c248943a5da0e10dfe90) commits. * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add panel for tracking rate of syscalls while writing or reading from disk via `process_io_(read|write)_syscalls_total` metrics. -* FEATURE: accept timestamps in milliseconds at `start`, `end` and `time` query args in [Prometheus querying API](./README.md#prometheus-querying-api-usage). See [these docs](./README.md#timestamp-formats) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459). -* FEATURE: [vmalert](./vmalert.md): update retry policy for pushing data to `-remoteWrite.url`. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed `-remoteWrite.retryMaxTime` (default is 30s). When retry time is exceeded vmalert drops the data dedicated for `-remoteWrite.url`. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See `-remoteWrite.retryMinInterval` and `-remoteWrite.retryMaxTime` cmd-line flags. -* FEATURE: [vmalert](./vmalert.md): expose `vmalert_remotewrite_send_duration_seconds_total` counter, which can be used for determining high saturation of every connection to remote storage with an alerting query `sum(rate(vmalert_remotewrite_send_duration_seconds_total[5m])) by(job, instance) > 0.9 * max(vmalert_remotewrite_concurrency) by(job, instance)`. This query triggers when a connection is saturated by more than 90%. This usually means that `-remoteWrite.concurrency` command-line flag must be increased in order to increase the number of concurrent writings into remote endpoint. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4516). -* FEATURE: [vmalert](./vmalert.md): display the error message received during unsuccessful config reload in vmalert's UI. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4076) for details. -* FEATURE: [vmalert](./vmalert.md): allow disabling of `step` param attached to [instant queries](./keyConcepts.md#instant-query). This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4573) for details. -* FEATURE: [vmalert](./vmalert.md): support option for "blackholing" alerting notifications if `-notifier.blackhole` cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4122) for details. Thanks to @venkatbvc for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4639). -* FEATURE: [vmalert](./vmalert.md): add unit test for alerting and recording rules, see more [details](./vmalert.md#unit-testing-for-rules) here. Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4596). -* FEATURE: [vmalert](./vmalert.md): allow overriding default GET params for rules with `graphite` datasource type, in the same way as it happens for `prometheus` type. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685). -* FEATURE: [vmalert](./vmalert.md): support `keep_firing_for` field for alerting rules. See docs updated [here](./vmalert.md#alerting-rules) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4529). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4669). -* FEATURE: [vmauth](./vmauth.md): expose `vmauth_user_request_duration_seconds` and `vmauth_unauthorized_user_request_duration_seconds` summary metrics for measuring requests latency per user. -* FEATURE: [vmbackup](./vmbackup.md): show backup progress percentage in log during backup uploading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4460). -* FEATURE: [vmrestore](./vmrestore.md): show restoring progress percentage in log during backup downloading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4460). +* FEATURE: accept timestamps in milliseconds at `start`, `end` and `time` query args in [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage). See [these docs](https://docs.victoriametrics.com/#timestamp-formats) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): update retry policy for pushing data to `-remoteWrite.url`. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed `-remoteWrite.retryMaxTime` (default is 30s). When retry time is exceeded vmalert drops the data dedicated for `-remoteWrite.url`. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See `-remoteWrite.retryMinInterval` and `-remoteWrite.retryMaxTime` cmd-line flags. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): expose `vmalert_remotewrite_send_duration_seconds_total` counter, which can be used for determining high saturation of every connection to remote storage with an alerting query `sum(rate(vmalert_remotewrite_send_duration_seconds_total[5m])) by(job, instance) > 0.9 * max(vmalert_remotewrite_concurrency) by(job, instance)`. This query triggers when a connection is saturated by more than 90%. This usually means that `-remoteWrite.concurrency` command-line flag must be increased in order to increase the number of concurrent writings into remote endpoint. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4516). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): display the error message received during unsuccessful config reload in vmalert's UI. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4076) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow disabling of `step` param attached to [instant queries](https://docs.victoriametrics.com/keyconcepts/#instant-query). This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4573) for details. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support option for "blackholing" alerting notifications if `-notifier.blackhole` cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4122) for details. Thanks to @venkatbvc for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4639). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add unit test for alerting and recording rules, see more [details](https://docs.victoriametrics.com/vmalert/#unit-testing-for-rules) here. Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4596). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow overriding default GET params for rules with `graphite` datasource type, in the same way as it happens for `prometheus` type. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support `keep_firing_for` field for alerting rules. See docs updated [here](https://docs.victoriametrics.com/vmalert/#alerting-rules) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4529). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4669). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): expose `vmauth_user_request_duration_seconds` and `vmauth_unauthorized_user_request_duration_seconds` summary metrics for measuring requests latency per user. +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): show backup progress percentage in log during backup uploading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4460). +* FEATURE: [vmrestore](https://docs.victoriametrics.com/vmrestore/): show restoring progress percentage in log during backup downloading. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4460). * FEATURE: add ability to fine-tune Graphite API limits via the following command-line flags: - `-search.maxGraphiteTagKeys` for limiting the number of tag keys returned from [Graphite API for tags](./README.md#graphite-tags-api-usage) - `-search.maxGraphiteTagValues` for limiting the number of tag values returned from [Graphite API for tag values](./README.md#graphite-tags-api-usage) - `-search.maxGraphiteSeries` for limiting the number of series (aka paths) returned from [Graphite API for series](./README.md#graphite-tags-api-usage) + `-search.maxGraphiteTagKeys` for limiting the number of tag keys returned from [Graphite API for tags](https://docs.victoriametrics.com/#graphite-tags-api-usage) + `-search.maxGraphiteTagValues` for limiting the number of tag values returned from [Graphite API for tag values](https://docs.victoriametrics.com/#graphite-tags-api-usage) + `-search.maxGraphiteSeries` for limiting the number of series (aka paths) returned from [Graphite API for series](https://docs.victoriametrics.com/#graphite-tags-api-usage) See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339). -* BUGFIX: properly return series from [/api/v1/series](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage) if it finds more than the `limit` series (`limit` is an optional query arg passed to this API). Previously the `limit exceeded error` error was returned in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841#issuecomment-1560055631). -* BUGFIX: [vmui](./README.md#vmui): fix application routing issues and problems with manual URL changes. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4408) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604). -* BUGFIX: add validation for invalid [partial RFC3339 timestamp formats](./Single-Server-VictoriaMetrics.md#timestamp-formats) in query and export APIs. -* BUGFIX: [vmctl](./vmctl.md): interrupt explore procedure in influx mode if vmctl found no numeric fields. -* BUGFIX: [vmctl](./vmctl.md): fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4553). -* BUGFIX: [vmctl](./vmctl.md): fix formatting issue, which could add superfluous `s` characters at the end of `samples/s` output during data migration. For example, it could write `samples/ssssss`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4555). -* BUGFIX: [vmalert](./vmalert.md): use RFC3339 time format in query args instead of unix timestamp for all issued queries to Prometheus-like datasources. -* BUGFIX: [vmalert](./vmalert.md): correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. -* BUGFIX: [vmalert](./vmalert.md): reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. +* BUGFIX: properly return series from [/api/v1/series](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage) if it finds more than the `limit` series (`limit` is an optional query arg passed to this API). Previously the `limit exceeded error` error was returned in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841#issuecomment-1560055631). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix application routing issues and problems with manual URL changes. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4408) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604). +* BUGFIX: add validation for invalid [partial RFC3339 timestamp formats](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats) in query and export APIs. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): interrupt explore procedure in influx mode if vmctl found no numeric fields. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4553). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix formatting issue, which could add superfluous `s` characters at the end of `samples/s` output during data migration. For example, it could write `samples/ssssss`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4555). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): use RFC3339 time format in query args instead of unix timestamp for all issued queries to Prometheus-like datasources. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. * BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch. -* BUGFIX: [vmagent](./vmagent.md): close HTTP connections to [service discovery](./sd_configs.md) servers when they are no longer needed. This should prevent from possible connection exhaustion in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724). -* BUGFIX: [vmagent](./vmagent.md): do not show [relabel debug](./vmagent.md#relabel-debug) links at the `/targets` page when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag, since it has no the original labels needed for relabel debug. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597). -* BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via [pushgateway protocol](./README.md#how-to-import-data-in-prometheus-exposition-format). This fixes Prometheus golang client compatibility. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly parse binary operations with reserved words on the right side such as `foo + (on{bar="baz"})`. Previously such queries could lead to panic. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4422). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): close HTTP connections to [service discovery](https://docs.victoriametrics.com/sd_configs/) servers when they are no longer needed. This should prevent from possible connection exhaustion in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not show [relabel debug](https://docs.victoriametrics.com/vmagent/#relabel-debug) links at the `/targets` page when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag, since it has no the original labels needed for relabel debug. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597). +* BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via [pushgateway protocol](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format). This fixes Prometheus golang client compatibility. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly parse binary operations with reserved words on the right side such as `foo + (on{bar="baz"})`. Previously such queries could lead to panic. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4422). * BUGFIX: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): display cache usage for all components on panel `Cache usage % by type` for cluster dashboard. Before, only vmstorage caches were shown. ## [v1.91.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.3) @@ -469,15 +469,15 @@ Released at 2023-06-30 * SECURITY: upgrade Go builder from Go1.20.4 to Go1.20.5. See [the list of issues addressed in Go1.20.5](https://github.com/golang/go/issues?q=milestone%3AGo1.20.5+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): fix possible panic at shutdown when [stream aggregation](./stream-aggregation.md) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4407) for details. -* BUGFIX: [vmagent](./vmagent.md): fixed service name detection for [consulagent service discovery](./sd_configs.md#consulagent_sd_configs) in case of a difference in service name and service id. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4390) for details. -* BUGFIX: [vmalert](./vmalert.md): retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request. -* BUGFIX: [vmalert](./vmalert.md): properly interrupt retry attempts on vmalert shutdown. Before, vmalert could have waited for all retries to finish for shutdown. -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix an issue with `vmbackupmanager` not being able to restore data from a backup stored in GCS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4420) for details. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly return error from [/api/v1/query](./keyConcepts.md#instant-query) and [/api/v1/query_range](./keyConcepts.md#range-query) at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` [limit](./Cluster-VictoriaMetrics.md#resource-usage-limits) is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472). -* BUGFIX: [storage](./Single-Server-VictoriaMetrics.md): prevent from possible crashloop after the migration from versions below `v1.90.0` to newer versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336) for details. -* BUGFIX: [vmui](./README.md#vmui): fix a memory leak issue associated with chart updates. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4455). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): fix removing storage data dir before restoring from backup. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible panic at shutdown when [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4407) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fixed service name detection for [consulagent service discovery](https://docs.victoriametrics.com/sd_configs/#consulagent_sd_configs) in case of a difference in service name and service id. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4390) for details. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly interrupt retry attempts on vmalert shutdown. Before, vmalert could have waited for all retries to finish for shutdown. +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix an issue with `vmbackupmanager` not being able to restore data from a backup stored in GCS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4420) for details. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly return error from [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` [limit](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472). +* BUGFIX: [storage](https://docs.victoriametrics.com/single-server-victoriametrics/): prevent from possible crashloop after the migration from versions below `v1.90.0` to newer versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336) for details. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a memory leak issue associated with chart updates. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4455). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix removing storage data dir before restoring from backup. * BUGFIX: vmselect: wait for all vmstorage nodes to respond when the `-replicationFactor` flag is set bigger than > 1. Before, vmselect could have [skip waiting for the slowest replicas](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711) to respond. This could have resulted in issues illustrated [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207). Now, this optimization is disabled by default and could be re-enabled by passing `-search.skipSlowReplicas` cmd-line flag to vmselect. See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4538). @@ -485,21 +485,21 @@ Released at 2023-06-30 Released at 2023-06-02 -* BUGFIX: [vmalert](./vmalert.md): fix nil map assignment panic in runtime introduced in this [change](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix nil map assignment panic in runtime introduced in this [change](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). ## [v1.91.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1) Released at 2023-06-01 -* FEATURE:[vmagent](./vmagent.md): Adds `follow_redirects` at service discovery level of scrape configuration. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4286). +* FEATURE:[vmagent](https://docs.victoriametrics.com/vmagent/): Adds `follow_redirects` at service discovery level of scrape configuration. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4286). * FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4364). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4366). -* BUGFIX: [vmalert](./vmalert.md): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). -* BUGFIX: [vmalert](./vmalert.md): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). -* BUGFIX: [vmauth](./vmauth.md): Properly handle LOCAL command for proxy protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335#issuecomment-1569864108). -* BUGFIX: [vmbackupmanager](./vmbackupmanager.md): Fixes crash on startup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4378). -* BUGFIX: [vmui](./README.md#vmui): fix bug with custom URL in global settings not respecting tenantID change. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4322). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): Properly handle LOCAL command for proxy protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335#issuecomment-1569864108). +* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): Fixes crash on startup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4378). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix bug with custom URL in global settings not respecting tenantID change. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4322). ## [v1.91.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.0) @@ -509,64 +509,64 @@ Released at 2023-05-18 * SECURITY: serve `/robots.txt` content to disallow indexing of the exposed instances by search engines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4128) for details. * FEATURE: update [docker compose environment](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#docker-compose-environment-for-victoriametrics) to V2 in respect to V1 deprecation notice from June 2023. See [Migrate to Compose V2](https://docs.docker.com/compose/migrate/). -* FEATURE: deprecate `-bigMergeConcurrency` command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for [background merges](./README.md#storage) can be controlled via `-smallMergeConcurrency` command-line flag, though it isn't recommended to change this flag in general case. +* FEATURE: deprecate `-bigMergeConcurrency` command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for [background merges](https://docs.victoriametrics.com/#storage) can be controlled via `-smallMergeConcurrency` command-line flag, though it isn't recommended to change this flag in general case. * FEATURE: do not execute the incoming request if it has been canceled by the client before the execution start. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4223). -* FEATURE: support time formats with timezones. For example, `2024-01-02+02:00` means `January 2, 2024` at `+02:00` time zone. See [these docs](./README.md#timestamp-formats). +* FEATURE: support time formats with timezones. For example, `2024-01-02+02:00` means `January 2, 2024` at `+02:00` time zone. See [these docs](https://docs.victoriametrics.com/#timestamp-formats). * FEATURE: expose `process_*` metrics at `/metrics` page of all the VictoriaMetrics components under Windows OS. See [this pull request](https://github.com/VictoriaMetrics/metrics/pull/47). * FEATURE: reduce the amounts of unimportant `INFO` logging during VictoriaMetrics startup / shutdown. This should improve visibility for potentially important logs. * FEATURE: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See [alpine 3.18.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): do not pollute logs with `cannot read hello: cannot read message with size 11: EOF` messages at `vmstorage` during TCP health checks performed by [Consul](https://developer.hashicorp.com/consul/docs/services/usage/checks) or [other services](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-health-check/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1762). -* FEATURE: [vmagent](./vmagent.md): support the ability to filter [consul_sd_configs](./sd_configs.md#consul_sd_configs) targets in more optimal way via new `filter` option. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183). -* FEATURE: [vmagent](./vmagent.md): add support for [consulagent_sd_configs](./sd_configs.md#consulagent_sd_configs). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3953). -* FEATURE: [vmagent](./vmagent.md): emit a warning if too small value is passed to `-remoteWrite.maxDiskUsagePerURL` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195). -* FEATURE: [vmalert](./vmalert.md): add support of recursive globs for `-rule` and `-rule.templates` command-line flags by using `**` in the glob pattern. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4041). -* FEATURE: [vmalert](./vmalert.md): add ability to specify custom per-group HTTP headers sent to the configured notifiers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4088). -* FEATURE: [vmalert](./vmalert.md): detect alerting rules which don't match any series. See [these docs](./vmalert.md#never-firing-alerts) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039). -* FEATURE: [vmalert](./vmalert.md): support loading rules via HTTP URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3352). Thanks to @Haleygo for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4212). -* FEATURE: [vmalert](./vmalert.md): add buttons for filtering groups/rules with errors or with no-match warning in web UI for page `/groups`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039). -* FEATURE: [vmalert](./vmalert.md): do not retry remote-write requests for responses with 4XX status codes. This aligns with [Prometheus remote write specification](https://prometheus.io/docs/concepts/remote_write_spec/). Thanks to @MichaHoffmann for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4134). -* FEATURE: [vmauth](./vmauth.md): add ability to filter incoming requests by IP. See [these docs](./vmauth.md#ip-filters) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491). -* FEATURE: [vmauth](./vmauth.md): add ability to proxy requests to the specified backends for unauthorized users. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4083). -* FEATURE: [vmauth](./vmauth.md): add ability to specify default route for unmatched requests. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084). -* FEATURE: [vmauth](./vmauth.md): retry `POST` requests on the remaining backends if the currently selected backend isn't reachable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4242). -* FEATURE: [vmui](./README.md#vmui): add ability to compare the data for the previous day with the data for the current day at [Cardinality Explorer](./README.md#cardinality-explorer). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3967). -* FEATURE: [vmui](./README.md#vmui): display histograms as heatmaps in [Metrics explorer](./README.md#metrics-explorer). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4111). -* FEATURE: [vmui](./README.md#vmui): add `WITH template` playground. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3811). -* FEATURE: [vmui](./README.md#vmui): add ability to debug relabeling. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3807). -* FEATURE: [vmui](./README.md#vmui): add an ability to copy and execute queries listed at [top queries](./README.md#top-queries) page. Also make more human readable the query duration column. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4292) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4299). -* FEATURE: [vmui](./README.md#vmui): increase default font size for better readability. -* FEATURE: [vmui](./README.md#vmui): [cardinality explorer](./README.md#cardinality-explorer): return back a table with labels containing the highest number of unique label values. See [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4213). -* FEATURE: [vmui](./README.md#vmui): add notification icon for queries that do not match any time series. A warning icon appears next to the query field when the executed query does not match any time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4211). -* FEATURE: [vmbackup](./vmbackup.md): add `-s3StorageClass` command-line flag for setting the storage class for AWS S3 backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4164). Thanks to @justcompile for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4166). -* FEATURE: [vmbackup](./vmbackup.md): store backup creation and completion time in `backup_complete.ignore` file of backup contents. This allows determining the exact timestamp when the backup was created and completed. -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): add `created_at` field to the output of `/api/v1/backups` API and `vmbackupmanager backup list` command. See this [doc](./vmbackupmanager.md#api-methods) for data format details. -* FEATURE: [vmbackupmanager](./vmbackupmanager.md): add commands for locking/unlocking backups against deletion by retention policy. See this [doc](./vmbackupmanager.md#api-methods) for data format details. -* FEATURE: [vmctl](./vmctl.md): add support for [different time formats](./Single-Server-VictoriaMetrics.md#timestamp-formats) for `--vm-native-filter-time-start` and `--vm-native-filter-time-end` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4091). -* FEATURE: [vmctl](./vmctl.md): set default value for `--vm-native-step-interval` command-line flag to `month`. This enables [time-based chunking](./vmctl.md#using-time-based-chunking-of-migration) of data based on monthly step value when using native migration mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4309). +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): do not pollute logs with `cannot read hello: cannot read message with size 11: EOF` messages at `vmstorage` during TCP health checks performed by [Consul](https://developer.hashicorp.com/consul/docs/services/usage/checks) or [other services](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-health-check/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1762). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support the ability to filter [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs) targets in more optimal way via new `filter` option. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [consulagent_sd_configs](https://docs.victoriametrics.com/sd_configs/#consulagent_sd_configs). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3953). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): emit a warning if too small value is passed to `-remoteWrite.maxDiskUsagePerURL` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add support of recursive globs for `-rule` and `-rule.templates` command-line flags by using `**` in the glob pattern. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4041). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to specify custom per-group HTTP headers sent to the configured notifiers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4088). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): detect alerting rules which don't match any series. See [these docs](https://docs.victoriametrics.com/vmalert/#never-firing-alerts) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support loading rules via HTTP URL. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3352). Thanks to @Haleygo for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4212). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add buttons for filtering groups/rules with errors or with no-match warning in web UI for page `/groups`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): do not retry remote-write requests for responses with 4XX status codes. This aligns with [Prometheus remote write specification](https://prometheus.io/docs/concepts/remote_write_spec/). Thanks to @MichaHoffmann for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4134). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to filter incoming requests by IP. See [these docs](https://docs.victoriametrics.com/vmauth/#ip-filters) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to proxy requests to the specified backends for unauthorized users. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4083). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to specify default route for unmatched requests. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): retry `POST` requests on the remaining backends if the currently selected backend isn't reachable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4242). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to compare the data for the previous day with the data for the current day at [Cardinality Explorer](https://docs.victoriametrics.com/#cardinality-explorer). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3967). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): display histograms as heatmaps in [Metrics explorer](https://docs.victoriametrics.com/#metrics-explorer). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4111). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `WITH template` playground. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3811). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to debug relabeling. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3807). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add an ability to copy and execute queries listed at [top queries](https://docs.victoriametrics.com/#top-queries) page. Also make more human readable the query duration column. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4292) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4299). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): increase default font size for better readability. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer): return back a table with labels containing the highest number of unique label values. See [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4213). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add notification icon for queries that do not match any time series. A warning icon appears next to the query field when the executed query does not match any time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4211). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): add `-s3StorageClass` command-line flag for setting the storage class for AWS S3 backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4164). Thanks to @justcompile for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4166). +* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/): store backup creation and completion time in `backup_complete.ignore` file of backup contents. This allows determining the exact timestamp when the backup was created and completed. +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): add `created_at` field to the output of `/api/v1/backups` API and `vmbackupmanager backup list` command. See this [doc](https://docs.victoriametrics.com/vmbackupmanager/#api-methods) for data format details. +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): add commands for locking/unlocking backups against deletion by retention policy. See this [doc](https://docs.victoriametrics.com/vmbackupmanager/#api-methods) for data format details. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add support for [different time formats](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats) for `--vm-native-filter-time-start` and `--vm-native-filter-time-end` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4091). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): set default value for `--vm-native-step-interval` command-line flag to `month`. This enables [time-based chunking](https://docs.victoriametrics.com/vmctl/#using-time-based-chunking-of-migration) of data based on monthly step value when using native migration mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4309). * BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores. * BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph `sum(rate(vm_slow_per_day_index_inserts_total))`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4258). * BUGFIX: fix possible infinite loop during `indexdb` rotation when `-retentionTimezoneOffset` command-line flag is set and the local timezone is not UTC. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207). Thanks to @faceair for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206). -* BUGFIX: do not panic at Windows during [snapshot deletion](./README.md#how-to-work-with-snapshots). Instead, delete the snapshot on the next restart. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183) for details. +* BUGFIX: do not panic at Windows during [snapshot deletion](https://docs.victoriametrics.com/#how-to-work-with-snapshots). Instead, delete the snapshot on the next restart. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183) for details. * BUGFIX: change the max allowed value for `-memory.allowedPercent` from 100 to 200. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4171). -* BUGFIX: properly limit the number of [OpenTSDB HTTP](./README.md#sending-opentsdb-data-via-http-apiput-requests) concurrent requests specified via `-maxConcurrentInserts` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204). Thanks to @zouxiang1993 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4208). -* BUGFIX: do not ignore trailing empty field in CSV lines when [importing data in CSV format](./README.md#how-to-import-csv-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048). +* BUGFIX: properly limit the number of [OpenTSDB HTTP](https://docs.victoriametrics.com/#sending-opentsdb-data-via-http-apiput-requests) concurrent requests specified via `-maxConcurrentInserts` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204). Thanks to @zouxiang1993 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4208). +* BUGFIX: do not ignore trailing empty field in CSV lines when [importing data in CSV format](https://docs.victoriametrics.com/#how-to-import-csv-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048). * BUGFIX: disallow `"` chars when parsing Prometheus label names, since they aren't allowed by [Prometheus text exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-example). Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as `foo{"bar"="baz"}` or `{foo:"bar",baz="aaa"}`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): prevent from possible panic when the number of vmstorage nodes increases when [automatic vmstorage discovery](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery) is enabled. -* BUGFIX: [MetricsQL](./MetricsQL.md): fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric[5M])` means rate over 5 minutes, 5 months or 5 million seconds. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120) issues. -* BUGFIX: [vmagent](./vmagent.md): properly handle the `vm_promscrape_config_last_reload_successful` metric after config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4260). -* BUGFIX: [vmagent](./vmagent.md): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue. -* BUGFIX: [vmalert](./vmalert.md): retry failed read request on the closed connection one more time. This improves rules execution reliability when connection between vmalert and datasource closes unexpectedly. -* BUGFIX: [vmalert](./vmalert.md): properly display an error when using `query` function for templating value of `-external.alert.source` flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4181). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). -* BUGFIX: [vmauth](./vmauth.md): do not return invalid auth credentials in http response by default, since it may be logged by client. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188). -* BUGFIX: [vmui](./README.md#vmui): fix the display of the tenant selector. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4160). -* BUGFIX: [vmui](./README.md#vmui): fix UI freeze when the query returns non-histogram series alongside histogram series. -* BUGFIX: [vmui](./README.md#vmui): fix the text display on buttons in Safari 16.4. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent from possible panic when the number of vmstorage nodes increases when [automatic vmstorage discovery](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) is enabled. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric[5M])` means rate over 5 minutes, 5 months or 5 million seconds. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120) issues. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly handle the `vm_promscrape_config_last_reload_successful` metric after config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4260). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): retry failed read request on the closed connection one more time. This improves rules execution reliability when connection between vmalert and datasource closes unexpectedly. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly display an error when using `query` function for templating value of `-external.alert.source` flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4181). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): do not return invalid auth credentials in http response by default, since it may be logged by client. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the display of the tenant selector. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4160). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix UI freeze when the query returns non-histogram series alongside histogram series. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the text display on buttons in Safari 16.4. * BUGFIX: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): update threshold for `TooHighMemoryUsage` alert from 90% to 80%, since 90% is too high for production environments. -* BUGFIX: [vmbackup](./vmbackup.md): fix compatibility with Windows OS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70). -* BUGFIX: [vmctl](./vmctl.md): fix performance issue when migrating data from VictoriaMetrics according to [these docs](./vmctl.md#migrating-data-from-victoriametrics). Add the ability to speed up the data migration via `--vm-native-disable-retries` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092). -* BUGFIX: [stream aggregation](./stream-aggregation.md): fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix compatibility with Windows OS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix performance issue when migrating data from VictoriaMetrics according to [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). Add the ability to speed up the data migration via `--vm-native-disable-retries` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277). ## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0) @@ -579,96 +579,96 @@ created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or n * SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See [alpine 3.17.3 release notes](https://alpinelinux.org/posts/Alpine-3.17.3-released.html). * SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See [the list of issues addressed in Go1.20.3](https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved). -* FEATURE: open source [Graphite Render API](./README.md#graphite-render-api-usage). This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See [this case study](./CaseStudies.md#grammarly) as an example. -* FEATURE: release Windows binaries for [single-node VictoriaMetrics](./README.md), [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md), [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70) issues. This release of VictoriaMetrics for Windows cannot delete [snapshots](./README.md#how-to-work-with-snapshots) due to Windows constraints. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183) for details. This issue should be resolved in future releases. +* FEATURE: open source [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See [this case study](https://docs.victoriametrics.com/casestudies/#grammarly) as an example. +* FEATURE: release Windows binaries for [single-node VictoriaMetrics](https://docs.victoriametrics.com/), [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/), [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236), [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70) issues. This release of VictoriaMetrics for Windows cannot delete [snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots) due to Windows constraints. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183) for details. This issue should be resolved in future releases. * FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds `-maxLabelValueLen`. This should simplify debugging for this case. -* FEATURE: [vmagent](./vmagent.md): show target URL when debugging [target relabeling](./vmagent.md#relabel-debug). This should simplify target relabel debugging a bit. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3882). -* FEATURE: [vmagent](./vmagent.md): add support for [VictoriaMetrics remote write protocol](./vmagent.md#victoriametrics-remote-write-protocol) when [sending / receiving data to / from Kafka](./vmagent.md#kafka-integration). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225). -* FEATURE: [vmagent](./vmagent.md): add `-kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957). -* FEATURE: [vmagent](./vmagent.md): add support for [Kafka producer and consumer](./vmagent.md#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271). -* FEATURE: [vmagent](./vmagent.md): delete unused buffered data at `-remoteWrite.tmpDataPath` directory when there is no matching `-remoteWrite.url` to send this data to. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014). -* FEATURE: [vmagent](./vmagent.md): add the ability for hot reloading of [stream aggregation](./stream-aggregation.md) configs. See [these docs](./stream-aggregation.md#configuration-update) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3969). -* FEATURE: check the contents of `-relabelConfig` and `-streamAggr.config` files additionally to `-promscrape.config` when single-node VictoriaMetrics runs with `-dryRun` command-line flag. This aligns the behaviour of single-node VictoriaMetrics with [vmagent](./vmagent.md) behaviour for `-dryRun` command-line flag. -* FEATURE: [vmui](./README.md#vmui): automatically draw a heatmap graph when the query selects a single [histogram](./keyConcepts.md#histogram). This simplifies analyzing histograms. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384). -* FEATURE: [vmui](./README.md#vmui): add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3971). -* FEATURE: [vmui](./README.md#vmui): hide messages longer than 3 lines in the trace. You can view the full message by clicking on the `show more` button. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3971). -* FEATURE: [vmui](./README.md#vmui): add the ability to manually input date and time when selecting a time range. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3968). -* FEATURE: [vmui](./README.md#vmui): updated usability and the search process in cardinality explorer. Made this process straightforward for user. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3986). -* FEATURE: [vmui](./README.md#vmui): add the ability to collapse/expand the legend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4045). -* FEATURE: [vmui](./README.md#vmui): add tips for working with the graph and legend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4045). -* FEATURE: [vmui](./README.md#vmui): add `apply` and `cancel` buttons to settings popup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4013). -* FEATURE: [vmctl](./vmctl.md): automatically disable progress bar when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3823). -* FEATURE: [vmauth](./vmauth.md): add `-configCheckInterval` command-line flag, which can be used for automatic re-reading the `-auth.config` file. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3990). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): show target URL when debugging [target relabeling](https://docs.victoriametrics.com/vmagent/#relabel-debug). This should simplify target relabel debugging a bit. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3882). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol) when [sending / receiving data to / from Kafka](https://docs.victoriametrics.com/vmagent/#kafka-integration). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [Kafka producer and consumer](https://docs.victoriametrics.com/vmagent/#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): delete unused buffered data at `-remoteWrite.tmpDataPath` directory when there is no matching `-remoteWrite.url` to send this data to. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add the ability for hot reloading of [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configs. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#configuration-update) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3969). +* FEATURE: check the contents of `-relabelConfig` and `-streamAggr.config` files additionally to `-promscrape.config` when single-node VictoriaMetrics runs with `-dryRun` command-line flag. This aligns the behaviour of single-node VictoriaMetrics with [vmagent](https://docs.victoriametrics.com/vmagent/) behaviour for `-dryRun` command-line flag. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): automatically draw a heatmap graph when the query selects a single [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram). This simplifies analyzing histograms. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3971). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): hide messages longer than 3 lines in the trace. You can view the full message by clicking on the `show more` button. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3971). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to manually input date and time when selecting a time range. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3968). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): updated usability and the search process in cardinality explorer. Made this process straightforward for user. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3986). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to collapse/expand the legend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4045). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add tips for working with the graph and legend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4045). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `apply` and `cancel` buttons to settings popup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4013). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): automatically disable progress bar when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3823). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add `-configCheckInterval` command-line flag, which can be used for automatic re-reading the `-auth.config` file. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3990). -* BUGFIX: prevent from slow [snapshot creating](./README.md#how-to-work-with-snapshots) under high data ingestion rate. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). -* BUGFIX: [vmauth](./vmauth.md): suppress [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt) parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. -* BUGFIX: [vmui](./README.md#vmui): fix displaying errors for each query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3987). -* BUGFIX: [vmbackup](./vmbackup.md): fix snapshot not being deleted in case of error during backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2055). -* BUGFIX: [stream aggregation](./stream-aggregation.md): suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at [vmagent](./vmagent.md) or when `-streamAggr.dedupInterval` command-line flag is set at [single-node VictoriaMetrics](./README.md). -* BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See [these docs](./README.md#environment-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3999). -* BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in [v1.86.0](./CHANGELOG.md#v1860). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). -* BUGFIX: [vmagent](./vmagent.md): fix CPU and memory usage spikes when files pointed by [file_sd_config](./sd_configs.md#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). +* BUGFIX: prevent from slow [snapshot creating](https://docs.victoriametrics.com/#how-to-work-with-snapshots) under high data ingestion rate. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): suppress [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt) parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix displaying errors for each query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3987). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix snapshot not being deleted in case of error during backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2055). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at [vmagent](https://docs.victoriametrics.com/vmagent/) or when `-streamAggr.dedupInterval` command-line flag is set at [single-node VictoriaMetrics](https://docs.victoriametrics.com/). +* BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See [these docs](https://docs.victoriametrics.com/#environment-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3999). +* BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in [v1.86.0](https://docs.victoriametrics.com/changelog/#v1860). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix CPU and memory usage spikes when files pointed by [file_sd_config](https://docs.victoriametrics.com/sd_configs/#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). * BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023). -* BUGFIX: properly support comma-separated filters inside [retention filters](./README.md#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). +* BUGFIX: properly support comma-separated filters inside [retention filters](https://docs.victoriametrics.com/#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). * BUGFIX: verify response code when fetching configuration files via HTTP. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034). -* BUGFIX: [vmalert](./vmalert.md): replace empty labels with `""` instead of `""` during templating, as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012). -* BUGFIX: [vmctl](./vmctl.md): properly pass multiple filters from `--vm-native-filter-match` command-line flag to the data source. Previously filters from `--vm-native-filter-match` were only used to discover the metric names, and the metric names like `__name__="metric_name"` has been taken into account, while the remaining filters were ignored. For example `--vm-native-src-addr={foo="bar",baz="abc"}` may found `metric_name{foo="bar",baz="abc"}` and filter was treated as `--vm-native-src-addr={__name__="metrics_name"}`, e.g. `foo="bar",baz="abc"` filter was ignored. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4062). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): replace empty labels with `""` instead of `""` during templating, as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): properly pass multiple filters from `--vm-native-filter-match` command-line flag to the data source. Previously filters from `--vm-native-filter-match` were only used to discover the metric names, and the metric names like `__name__="metric_name"` has been taken into account, while the remaining filters were ignored. For example `--vm-native-src-addr={foo="bar",baz="abc"}` may found `metric_name{foo="bar",baz="abc"}` and filter was treated as `--vm-native-src-addr={__name__="metrics_name"}`, e.g. `foo="bar",baz="abc"` filter was ignored. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4062). ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1) Released at 2023-03-12 -* BUGFIX: prevent from possible `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade to [v1.89.0](./CHANGELOG.md#v1890). +* BUGFIX: prevent from possible `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade to [v1.89.0](https://docs.victoriametrics.com/changelog/#v1890). ## [v1.89.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.0) Released at 2023-03-12 **Update note: this release can crash with `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade from the previous releases. -This issue can be fixed by removing caches stored on disk according to [these docs](./README.md#cache-removal). -Another option is to upgrade to [v1.89.1](./CHANGELOG.md#v1891).** +This issue can be fixed by removing caches stored on disk according to [these docs](https://docs.victoriametrics.com/#cache-removal). +Another option is to upgrade to [v1.89.1](https://docs.victoriametrics.com/changelog/#v1891).** * SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See [the list of issues addressed in Go1.20.2](https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved). -* FEATURE: [vmctl](./vmctl.md): increase the default value for `--remote-read-http-timeout` command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3879). -* FEATURE: [vmctl](./vmctl.md): migrate series one-by-one in [vm-native mode](./vmctl.md#migrating-data-from-victoriametrics). This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3859) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600). -* FEATURE: [vmctl](./vmctl.md): add `--vm-native-src-headers` and `--vm-native-dst-headers` command-line flags, which can be used for setting custom HTTP headers during [vm-native migration mode](./vmctl.md#migrating-data-from-victoriametrics). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3906). -* FEATURE: [vmctl](./vmctl.md): add `--vm-native-src-bearer-token` and `--vm-native-dst-bearer-token` command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during [vm-native migration mode](./vmctl.md#migrating-data-from-victoriametrics). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3835). -* FEATURE: [vmctl](./vmctl.md): add `--vm-native-disable-http-keep-alive` command-line flag to allow `vmctl` to use non-persistent HTTP connections in [vm-native migration mode](./vmctl.md#migrating-data-from-victoriametrics). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3909). -* FEATURE: [vmalert](./vmalert.md): log number of configration files found for each specified `-rule` command-line flag. -* FEATURE: [vmalert enterprise](./vmalert.md): concurrently [read config files from S3, GCS or S3-compatible object storage](./vmalert.md#reading-rules-from-object-storage). This significantly improves config load speed for cases when there are thousands of files to read from the object storage. +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): increase the default value for `--remote-read-http-timeout` command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3879). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): migrate series one-by-one in [vm-native mode](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3859) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--vm-native-src-headers` and `--vm-native-dst-headers` command-line flags, which can be used for setting custom HTTP headers during [vm-native migration mode](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3906). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--vm-native-src-bearer-token` and `--vm-native-dst-bearer-token` command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during [vm-native migration mode](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3835). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--vm-native-disable-http-keep-alive` command-line flag to allow `vmctl` to use non-persistent HTTP connections in [vm-native migration mode](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). Thanks to @baconmania for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3909). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): log number of configration files found for each specified `-rule` command-line flag. +* FEATURE: [vmalert enterprise](https://docs.victoriametrics.com/vmalert/): concurrently [read config files from S3, GCS or S3-compatible object storage](https://docs.victoriametrics.com/vmalert/#reading-rules-from-object-storage). This significantly improves config load speed for cases when there are thousands of files to read from the object storage. * BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3946). * BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692). -* BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during [background merge](./README.md#storage). The issue has been introduced at [v1.85.0](./CHANGELOG.md#v1850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897). +* BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during [background merge](https://docs.victoriametrics.com/#storage). The issue has been introduced at [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897). * BUGFIX: prevent from possible `SIGBUS` crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for [the initial attempt to fix it](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](./CHANGELOG.md#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). -* BUGFIX: [vmagent](./vmagent.md): fix panic when [writing data to Kafka](./vmagent.md#writing-metrics-to-kafka). The panic has been introduced in [v1.88.0](./CHANGELOG.md#v1880). -* BUGFIX: [vmui](./README.md#vmui): stop showing `Please enter a valid Query and execute it` error message on the first load of vmui. -* BUGFIX: [vmui](./README.md#vmui): properly process `Run in VMUI` button click in [VictoriaMetrics datasource plugin for Grafana](https://github.com/VictoriaMetrics/victoriametrics-datasource). -* BUGFIX: [vmui](./README.md#vmui): fix the display of the selected value for dropdowns on `Explore` page. -* BUGFIX: [vmui](./README.md#vmui): do not send `step` param for instant queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896). -* BUGFIX: [vmauth](./vmauth.md): fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt)-encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at [v1.87.0](./CHANGELOG.md#v1870) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). -* BUGFIX: [vmgateway](./vmgateway.md): properly parse RSA public key discovered via JWK endpoint. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix panic when [writing data to Kafka](https://docs.victoriametrics.com/vmagent/#writing-metrics-to-kafka). The panic has been introduced in [v1.88.0](https://docs.victoriametrics.com/changelog/#v1880). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): stop showing `Please enter a valid Query and execute it` error message on the first load of vmui. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly process `Run in VMUI` button click in [VictoriaMetrics datasource plugin for Grafana](https://github.com/VictoriaMetrics/victoriametrics-datasource). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the display of the selected value for dropdowns on `Explore` page. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not send `step` param for instant queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt)-encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). +* BUGFIX: [vmgateway](https://docs.victoriametrics.com/vmgateway/): properly parse RSA public key discovered via JWK endpoint. ## [v1.88.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.88.1) Released at 2023-02-27 -* FEATURE: add `-snapshotCreateTimeout` flag to allow configuring timeout for [snapshot process](./README.md#how-to-work-with-snapshots). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). -* FEATURE: expose `vm_http_requests_total` and `vm_http_request_errors_total` metrics for `snapshot/*` [paths](./README.md#how-to-work-with-snapshots) at [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) `vmstorage` and [VictoriaMetrics Single](./Single-Server-VictoriaMetrics.md). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). -* FEATURE: [vmgateway](./vmgateway.md): add the ability to discover keys for JWT verification via [OpenID discovery endpoint](https://openid.net/specs/openid-connect-discovery-1_0.html). See [these docs](./vmgateway.md#using-openid-discovery-endpoint-for-jwt-signature-verification). +* FEATURE: add `-snapshotCreateTimeout` flag to allow configuring timeout for [snapshot process](https://docs.victoriametrics.com/#how-to-work-with-snapshots). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). +* FEATURE: expose `vm_http_requests_total` and `vm_http_request_errors_total` metrics for `snapshot/*` [paths](https://docs.victoriametrics.com/#how-to-work-with-snapshots) at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) `vmstorage` and [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add the ability to discover keys for JWT verification via [OpenID discovery endpoint](https://openid.net/specs/openid-connect-discovery-1_0.html). See [these docs](https://docs.victoriametrics.com/vmgateway/#using-openid-discovery-endpoint-for-jwt-signature-verification). * FEATURE: add `-internStringDisableCache` command-line flag for disabling the cache for [interned strings](https://en.wikipedia.org/wiki/String_interning). This flag may be useful in [some cases](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863) for reducing memory usage at the cost of higher CPU usage. * FEATURE: add `-internStringCacheExpireDuration` command-line flag for controlling the lifetime of cached [interned strings](https://en.wikipedia.org/wiki/String_interning). -* BUGFIX: [MetricsQL](./MetricsQL.md): fix panic when executing the query `aggr_func(rollup*(some_value))`. The panic has been introduced in [v1.88.0](./CHANGELOG.md#v1880). -* BUGFIX: [vmagent](./vmagent.md): use the provided `-remoteWrite.*` auth options when determining whether the remote storage supports [VictoriaMetrics remote write protocol](./vmagent.md#victoriametrics-remote-write-protocol). Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol. -* BUGFIX: [vmagent](./vmagent.md): do not register `vm_promscrape_config_*` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. -* BUGFIX: [vmctl](./vmctl.md): skip measurements with no fields when migrating data from influxdb. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3837). -* BUGFIX: delete failed snapshot contents from disk on failed attempt to [create snapshot](./README.md#how-to-work-with-snapshots). Previously failed snapshot contents could remain on disk in incomplete state. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3858) +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): fix panic when executing the query `aggr_func(rollup*(some_value))`. The panic has been introduced in [v1.88.0](https://docs.victoriametrics.com/changelog/#v1880). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): use the provided `-remoteWrite.*` auth options when determining whether the remote storage supports [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not register `vm_promscrape_config_*` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): skip measurements with no fields when migrating data from influxdb. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3837). +* BUGFIX: delete failed snapshot contents from disk on failed attempt to [create snapshot](https://docs.victoriametrics.com/#how-to-work-with-snapshots). Previously failed snapshot contents could remain on disk in incomplete state. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3858) ## [v1.88.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.88.0) @@ -677,139 +677,139 @@ Released at 2023-02-24 * SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See [alpine 3.17.2 release notes](https://alpinelinux.org/posts/Alpine-3.17.2-released.html). * SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See [the list of issues addressed in Go1.20.1](https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved). -* FEATURE: [vmagent](./vmagent.md): add support for [VictoriaMetrics remote write protocol](./vmagent.md#victoriametrics-remote-write-protocol). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to VictoriaMetrics located in another datacenter or availability zone. This also allows reducing disk IO under high load when `vmagent` starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225). -* FEATURE: [vmagent](./vmagent.md): add support for [Kuma](http://kuma.io/) Control Plane targets discovery aka [kuma_sd_configs](./sd_configs.md#kuma_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389). -* FEATURE: [vmgateway](./vmgateway.md): add the ability to verify JWT signature via [JWKS endpoint](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets). See [these docs](./vmgateway.md#using-jwks-endpoint-for-jwt-signature-verification). -* FEATURE: [vmauth](./vmauth.md): add the ability to limit the number of concurrent requests on a per-user basis via `-maxConcurrentPerUserRequests` command-line flag and via `max_concurrent_requests` config option. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346) and [these docs](./vmauth.md#concurrency-limiting). -* FEATURE: [vmauth](./vmauth.md): automatically retry failing `GET` requests on all [the configured backends](./vmauth.md#load-balancing). Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends. -* FEATURE: [vmauth](./vmauth.md): choose the backend with the minimum number of concurrently executed requests [among the configured backends](./vmauth.md#load-balancing) in a round-robin manner for serving the incoming requests. This allows spreading the load among backends more evenly, while improving the response time. -* FEATURE: [vmalert enterprise](./vmalert.md): add ability to read alerting and recording rules from S3, GCS or S3-compatible object storage. See [these docs](./vmalert.md#reading-rules-from-object-storage). -* FEATURE: [vmctl](./vmctl.md): automatically retry requests to remote storage if up to 5 errors occur during the data migration process. This should help continuing the data migration process on temporary errors. Previously `vmctl` was stopping after the first error. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600). -* FEATURE: [MetricsQL](./MetricsQL.md): support optional 2nd argument `min`, `max` or `avg` for [rollup](./MetricsQL.md#rollup), [rollup_delta](./MetricsQL.md#rollup_delta), [rollup_deriv](./MetricsQL.md#rollup_deriv), [rollup_increase](./MetricsQL.md#rollup_increase), [rollup_rate](./MetricsQL.md#rollup_rate) and [rollup_scrape_interval](./MetricsQL.md#rollup_scrape_interval) function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, `rollup_rate(requests_total[1i], "max")` would return only the max increase rates for `requests_total` metric per each interval between adjacent points on the graph. See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) for details. -* FEATURE: [MetricsQL](./MetricsQL.md): support optional 2nd argument `open`, `low`, `high`, `close` for [rollup_candlestick](./MetricsQL.md#rollup_candlestick) function. If the second argument is passed, then the function returns only the selected aggregation type. -* FEATURE: [MetricsQL](./MetricsQL.md): add [share(q)](./MetricsQL.md#share) aggregate function. -* FEATURE: [MetricsQL](./MetricsQL.md): add `mad_over_time(m[d])` function for calculating the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) over raw samples on the lookbehind window `d`. See [this feature request](https://github.com/prometheus/prometheus/issues/5514). -* FEATURE: [MetricsQL](./MetricsQL.md): add `range_mad(q)` function for calculating the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) over points per each time series returned by `q`. -* FEATURE: [MetricsQL](./MetricsQL.md): add `range_zscore(q)` function for calculating [z-score](https://en.wikipedia.org/wiki/Standard_score) over points per each time series returned from `q`. -* FEATURE: [MetricsQL](./MetricsQL.md): add `range_trim_outliers(k, q)` function for dropping outliers located farther than `k*range_mad(q)` from the `range_median(q)`. This should help removing outliers during query time at [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759). -* FEATURE: [MetricsQL](./MetricsQL.md): add `range_trim_zscore(z, q)` function for dropping outliers located farther than `z*range_stddev(q)` from `range_avg(q)`. This should help removing outliers during query time at [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759). -* FEATURE: [vmui](./README.md#vmui): show `median` instead of `avg` in graph tooltip and line legend, since `median` is more tolerant against spikes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706). -* FEATURE: add `-search.maxSeriesPerAggrFunc` command-line flag, which can be used for limiting the number of time series [MetricsQL aggregate functions](./MetricsQL.md#aggregate-functions) can return in a single query. This flag can be useful for preventing OOMs when [count_values](./MetricsQL.md#count_values) function is improperly used. -* FEATURE: [vmui](./README.md#vmui): small UX improvements for mobile view. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3848). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to VictoriaMetrics located in another datacenter or availability zone. This also allows reducing disk IO under high load when `vmagent` starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [Kuma](http://kuma.io/) Control Plane targets discovery aka [kuma_sd_configs](https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389). +* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway/): add the ability to verify JWT signature via [JWKS endpoint](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets). See [these docs](https://docs.victoriametrics.com/vmgateway/#using-jwks-endpoint-for-jwt-signature-verification). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add the ability to limit the number of concurrent requests on a per-user basis via `-maxConcurrentPerUserRequests` command-line flag and via `max_concurrent_requests` config option. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346) and [these docs](https://docs.victoriametrics.com/vmauth/#concurrency-limiting). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): automatically retry failing `GET` requests on all [the configured backends](https://docs.victoriametrics.com/vmauth/#load-balancing). Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends. +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): choose the backend with the minimum number of concurrently executed requests [among the configured backends](https://docs.victoriametrics.com/vmauth/#load-balancing) in a round-robin manner for serving the incoming requests. This allows spreading the load among backends more evenly, while improving the response time. +* FEATURE: [vmalert enterprise](https://docs.victoriametrics.com/vmalert/): add ability to read alerting and recording rules from S3, GCS or S3-compatible object storage. See [these docs](https://docs.victoriametrics.com/vmalert/#reading-rules-from-object-storage). +* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): automatically retry requests to remote storage if up to 5 errors occur during the data migration process. This should help continuing the data migration process on temporary errors. Previously `vmctl` was stopping after the first error. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): support optional 2nd argument `min`, `max` or `avg` for [rollup](https://docs.victoriametrics.com/metricsql/#rollup), [rollup_delta](https://docs.victoriametrics.com/metricsql/#rollup_delta), [rollup_deriv](https://docs.victoriametrics.com/metricsql/#rollup_deriv), [rollup_increase](https://docs.victoriametrics.com/metricsql/#rollup_increase), [rollup_rate](https://docs.victoriametrics.com/metricsql/#rollup_rate) and [rollup_scrape_interval](https://docs.victoriametrics.com/metricsql/#rollup_scrape_interval) function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, `rollup_rate(requests_total[1i], "max")` would return only the max increase rates for `requests_total` metric per each interval between adjacent points on the graph. See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) for details. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): support optional 2nd argument `open`, `low`, `high`, `close` for [rollup_candlestick](https://docs.victoriametrics.com/metricsql/#rollup_candlestick) function. If the second argument is passed, then the function returns only the selected aggregation type. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add [share(q)](https://docs.victoriametrics.com/metricsql/#share) aggregate function. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `mad_over_time(m[d])` function for calculating the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) over raw samples on the lookbehind window `d`. See [this feature request](https://github.com/prometheus/prometheus/issues/5514). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `range_mad(q)` function for calculating the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) over points per each time series returned by `q`. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `range_zscore(q)` function for calculating [z-score](https://en.wikipedia.org/wiki/Standard_score) over points per each time series returned from `q`. +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `range_trim_outliers(k, q)` function for dropping outliers located farther than `k*range_mad(q)` from the `range_median(q)`. This should help removing outliers during query time at [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): add `range_trim_zscore(z, q)` function for dropping outliers located farther than `z*range_stddev(q)` from `range_avg(q)`. This should help removing outliers during query time at [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): show `median` instead of `avg` in graph tooltip and line legend, since `median` is more tolerant against spikes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706). +* FEATURE: add `-search.maxSeriesPerAggrFunc` command-line flag, which can be used for limiting the number of time series [MetricsQL aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions) can return in a single query. This flag can be useful for preventing OOMs when [count_values](https://docs.victoriametrics.com/metricsql/#count_values) function is improperly used. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): small UX improvements for mobile view. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3848). * FEATURE: add `-search.logQueryMemoryUsage` command-line flag for logging queries, which need more memory than specified by this command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553). Thanks to @michal-kralik for the idea and the initial implementation. * FEATURE: allow setting zero value for `-search.latencyOffset` command-line flag. This may be needed in [some cases](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836). Previously the minimum supported value for `-search.latencyOffset` command-line flag was `1s`. -* BUGFIX: [vmagent](./vmagent.md): immediately cancel in-flight scrape requests during configuration reload when [stream parsing mode](./vmagent.md#stream-parsing-mode) is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). -* BUGFIX: [vmagent](./vmagent.md): do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes [http keep-alive connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection) between scrapes. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747) issues. -* BUGFIX: [vmagent](./vmagent.md): fix [Azure service discovery](./sd_configs.md#azure_sd_configs) inside [Azure Container App](https://learn.microsoft.com/en-us/azure/container-apps/overview). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830). Thanks to @MattiasAng for the fix! +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): immediately cancel in-flight scrape requests during configuration reload when [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes [http keep-alive connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection) between scrapes. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747) issues. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix [Azure service discovery](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs) inside [Azure Container App](https://learn.microsoft.com/en-us/azure/container-apps/overview). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830). Thanks to @MattiasAng for the fix! * BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from `cannot create hard links from ...must-remove...` errors when making snapshots / backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3858). -* BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during [background merges of big parts](./README.md#storage) on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in [v1.85.0](./CHANGELOG.md#v1850) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790). -* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). -* BUGFIX: [MetricsQL](./MetricsQL.md): do not add extrapolated points outside the real points when using [interpolate()](./MetricsQL.md#interpolate) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3816). +* BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during [background merges of big parts](https://docs.victoriametrics.com/#storage) on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790). +* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): do not add extrapolated points outside the real points when using [interpolate()](https://docs.victoriametrics.com/metricsql/#interpolate) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3816). # [v1.87.12](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.12) Released at 2023-12-10 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See [alpine 3.19.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html). * SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See [the list of issues addressed in Go1.21.5](https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved). -* BUGFIX: [vmalert](./vmalert.md): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. -* BUGFIX: properly escape `<` character in responses returned via [`/federate`](./README.md#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. +* BUGFIX: properly escape `<` character in responses returned via [`/federate`](https://docs.victoriametrics.com/#federation) endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431). ## [v1.87.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.11) Released at 2023-11-14 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.21.3 to Go1.21.4. [the list of issues addressed in Go1.21.4](https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): properly apply [relabeling](./vmagent.md#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). -* BUGFIX: fix panic, which could occur when [query tracing](./README.md#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) with `regex`, which start and end with `.+` or `.*` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.*foo|bar|baz.*`. Previously such regexps were improperly parsed, which could result in unexpected relabeling results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297). +* BUGFIX: fix panic, which could occur when [query tracing](https://docs.victoriametrics.com/#query-tracing) is enabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5319). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details. ## [v1.87.10](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.10) Released at 2023-10-16 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See [the list of issues addressed in Go1.21.2](https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved) and [the list of issues addressed in Go1.21.3](https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved). -* BUGFIX: [storage](./Single-Server-VictoriaMetrics.md): prevent from livelock when [forced merge](./README.md#forced-merge) is called under high data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4987). -* BUGFIX: [Graphite Render API](./README.md#graphite-render-api-usage): correctly return `null` instead of `Inf` in JSON query responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3783). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): fix ingestion via [multitenant url](./Cluster-VictoriaMetrics.md#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.87.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.8). -* BUGFIX: [vmagent](./vmagent.md): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. +* BUGFIX: [storage](https://docs.victoriametrics.com/single-server-victoriametrics/): prevent from livelock when [forced merge](https://docs.victoriametrics.com/#forced-merge) is called under high data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4987). +* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): correctly return `null` instead of `Inf` in JSON query responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3783). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): fix ingestion via [multitenant url](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061). The bug has been introduced in [v1.87.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.8). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix support of legacy DataDog agent, which adds trailing slashes to urls. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5078). Thanks to @maxb for spotting the issue. ## [v1.87.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.9) Released at 2023-09-10 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.21.0 to Go1.21.1. See [the list of issues addressed in Go1.20.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved). -* BUGFIX: [vminsert enterprise](./enterprise.md): properly parse `/insert/multitenant/*` urls, which have been broken since [v1.93.2](#v1932). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4947). +* BUGFIX: [vminsert enterprise](https://docs.victoriametrics.com/enterprise/): properly parse `/insert/multitenant/*` urls, which have been broken since [v1.93.2](#v1932). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4947). * BUGFIX: properly build production armv5 binaries for `GOARCH=arm`. This has been broken after the upgrading of Go builder to Go1.21.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965). -* BUGFIX: [vmselect](./Cluster-VictoriaMetrics.md): return `503 Service Unavailable` status code when [partial responses](./Cluster-VictoriaMetrics.md#cluster-availability) are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entity` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. -* BUGFIX: [vmalert](./vmalert.md): fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/eccecdf177115297fa1dc4d42d38e23de9a9f2cb) starting from [v1.87.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.7). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908). +* BUGFIX: [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/): return `503 Service Unavailable` status code when [partial responses](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability) are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entity` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/eccecdf177115297fa1dc4d42d38e23de9a9f2cb) starting from [v1.87.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.7). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908). ## [v1.87.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.8) Released at 2023-09-01 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** -* BUGFIX: [build](./README.md): fix Docker builds for old Docker releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907). +* BUGFIX: [build](https://docs.victoriametrics.com/): fix Docker builds for old Docker releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907). * BUGFIX: vmselect: correctly handle requests with `/select/multitenant` prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): properly check for [read-only state](./Cluster-VictoriaMetrics.md#readonly-mode) at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). -* BUGFIX: [vminsert](./Cluster-VictoriaMetrics.md): properly close broken vmstorage connection during [read-only state](./Cluster-VictoriaMetrics.md#readonly-mode) checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). -* BUGFIX: [vmstorage](./Cluster-VictoriaMetrics.md): prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4932). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): properly check for [read-only state](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode) at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). +* BUGFIX: [vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/): properly close broken vmstorage connection during [read-only state](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode) checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870). +* BUGFIX: [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4932). * BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form `-boolFlagName value`, since this leads to silent incomplete flags' parsing. This form should be replaced with `-boolFlagName=value`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4845). * BUGFIX: properly replace `:` chars in label names with `_` when `-usePromCompatibleNaming` command-line flag is passed to `vmagent`, `vminsert` or single-node VictoriaMetrics. This addresses [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113#issuecomment-1275077071). -* BUGFIX: [vmbackup](./vmbackup.md): correctly check if specified `-dst` belongs to specified `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837). +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): correctly check if specified `-dst` belongs to specified `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837). ## [v1.87.7](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.7) Released at 2023-08-12 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.20.4 to Go1.21.0. * SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See [alpine 3.18.3 release notes](https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html). * BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch. -* BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via [pushgateway protocol](./README.md#how-to-import-data-in-prometheus-exposition-format). This fixes Prometheus golang client compatibility. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692). -* BUGFIX: [vmagent](./vmagent.md): properly validate scheme for `proxy_url` field at the scrape config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4811) for details. -* BUGFIX: [vmagent](./vmagent.md): close HTTP connections to [service discovery](./sd_configs.md) servers when they are no longer needed. This should prevent from possible connection exhaustion in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724). -* BUGFIX: [vmagent](./vmagent.md): properly apply `if` filters during [relabeling](./vmagent.md#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). -* BUGFIX: [vmagent](./vmagent.md): fix possible panic at shutdown when [stream aggregation](./stream-aggregation.md) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4407) for details. -* BUGFIX: [vmagent](./vmagent.md): use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at [scrape_config](./sd_configs.md#scrape_configs). This fixes gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) or similar exporters, which export metrics with invalid timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799) for details. -* BUGFIX: [vmauth](./vmauth.md): Properly handle LOCAL command for proxy protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335#issuecomment-1569864108). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly return error from [/api/v1/query](./keyConcepts.md#instant-query) and [/api/v1/query_range](./keyConcepts.md#range-query) at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` [limit](./Cluster-VictoriaMetrics.md#resource-usage-limits) is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472). -* BUGFIX: [vmalert](./vmalert.md): correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. -* BUGFIX: [vmalert](./vmalert.md): reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. -* BUGFIX: [vmalert](./vmalert.md): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). -* BUGFIX: [vmalert](./vmalert.md): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). -* BUGFIX: [vmctl](./vmctl.md): interrupt explore procedure in influx mode if vmctl found no numeric fields. -* BUGFIX: [vmctl](./vmctl.md): fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4553). +* BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via [pushgateway protocol](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format). This fixes Prometheus golang client compatibility. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly validate scheme for `proxy_url` field at the scrape config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4811) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): close HTTP connections to [service discovery](https://docs.victoriametrics.com/sd_configs/) servers when they are no longer needed. This should prevent from possible connection exhaustion in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply `if` filters during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). Previously the `if` filter could improperly work. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4816). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible panic at shutdown when [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4407) for details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). This fixes gaps for metrics collected from [cadvisor](https://github.com/google/cadvisor) or similar exporters, which export metrics with invalid timestamps. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799) for details. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): Properly handle LOCAL command for proxy protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335#issuecomment-1569864108). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly return error from [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` [limit](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): interrupt explore procedure in influx mode if vmctl found no numeric fields. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4553). ## [v1.87.6](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.6) Released at 2023-05-18 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See [the list of issues addressed in Go1.20.4](https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See [alpine 3.18.0 release notes](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html). @@ -817,214 +817,214 @@ The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHAN * BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores. * BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph `sum(rate(vm_slow_per_day_index_inserts_total))`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4258). -* BUGFIX: do not ignore trailing empty field in CSV lines when [importing data in CSV format](./README.md#how-to-import-csv-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048). +* BUGFIX: do not ignore trailing empty field in CSV lines when [importing data in CSV format](https://docs.victoriametrics.com/#how-to-import-csv-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048). * BUGFIX: disallow `"` chars when parsing Prometheus label names, since they aren't allowed by [Prometheus text exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-example). Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as `foo{"bar"="baz"}` or `{foo:"bar",baz="aaa"}`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284). -* BUGFIX: [MetricsQL](./MetricsQL.md): fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric[5M])` means rate over 5 minutes, 5 months or 5 million seconds. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120) issues. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): prevent from possible panic when the number of vmstorage nodes increases when [automatic vmstorage discovery](./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery) is enabled. -* BUGFIX: properly limit the number of [OpenTSDB HTTP](./README.md#sending-opentsdb-data-via-http-apiput-requests) concurrent requests specified via `-maxConcurrentInserts` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204). Thanks to @zouxiang1993 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4208). -* BUGFIX: [vmalert](./vmalert.md): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). -* BUGFIX: [vmagent](./vmagent.md): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue. +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric[5M])` means rate over 5 minutes, 5 months or 5 million seconds. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120) issues. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): prevent from possible panic when the number of vmstorage nodes increases when [automatic vmstorage discovery](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) is enabled. +* BUGFIX: properly limit the number of [OpenTSDB HTTP](https://docs.victoriametrics.com/#sending-opentsdb-data-via-http-apiput-requests) concurrent requests specified via `-maxConcurrentInserts` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204). Thanks to @zouxiang1993 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4208). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue. * BUGFIX: fix possible infinite loop during `indexdb` rotation when `-retentionTimezoneOffset` command-line flag is set and the local timezone is not UTC. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207). Thanks to @faceair for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206). -* BUGFIX: [vmauth](./vmauth.md): do not return invalid auth credentials in http response by default, since it may be logged by client. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): do not return invalid auth credentials in http response by default, since it may be logged by client. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188). * BUGFIX: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): update threshold for `TooHighMemoryUsage` alert from 90% to 80%, since 90% is too high for production environments. -* BUGFIX: [vmagent](./vmagent.md): properly handle the `vm_promscrape_config_last_reload_successful` metric after config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4260). -* BUGFIX: [stream aggregation](./stream-aggregation.md): fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277). -* BUGFIX: [stream aggregation](./stream-aggregation.md): suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at [vmagent](./vmagent.md) or when `-streamAggr.dedupInterval` command-line flag is set at [single-node VictoriaMetrics](./README.md). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly handle the `vm_promscrape_config_last_reload_successful` metric after config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4260). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277). +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at [vmagent](https://docs.victoriametrics.com/vmagent/) or when `-streamAggr.dedupInterval` command-line flag is set at [single-node VictoriaMetrics](https://docs.victoriametrics.com/). ## [v1.87.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.5) Released at 2023-04-06 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See [alpine 3.17.3 release notes](https://alpinelinux.org/posts/Alpine-3.17.3-released.html). * SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See [the list of issues addressed in Go1.20.3](https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). -* BUGFIX: [vmagent](./vmagent.md): fix CPU and memory usage spikes when files pointed by [file_sd_config](./sd_configs.md#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix CPU and memory usage spikes when files pointed by [file_sd_config](https://docs.victoriametrics.com/sd_configs/#file_sd_configs) cannot be re-read. See [this_issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989). * BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023). -* BUGFIX: properly support comma-separated filters inside [retention filters](./README.md#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). +* BUGFIX: properly support comma-separated filters inside [retention filters](https://docs.victoriametrics.com/#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). * BUGFIX: verify response code when fetching configuration files via HTTP. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034). ## [v1.87.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.4) Released at 2023-03-25 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** -* BUGFIX: prevent from slow [snapshot creating](./README.md#how-to-work-with-snapshots) under high data ingestion rate. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). -* BUGFIX: [vmauth](./vmauth.md): suppress [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt) parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. -* BUGFIX: [vmbackup](./vmbackup.md): fix snapshot not being deleted in case of error during backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2055). -* BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See [these docs](./README.md#environment-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3999). -* BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in [v1.86.0](./CHANGELOG.md#v1860). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966). +* BUGFIX: prevent from slow [snapshot creating](https://docs.victoriametrics.com/#how-to-work-with-snapshots) under high data ingestion rate. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): suppress [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt) parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix snapshot not being deleted in case of error during backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2055). +* BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See [these docs](https://docs.victoriametrics.com/#environment-variables) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3999). +* BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in [v1.86.0](https://docs.victoriametrics.com/changelog/#v1860). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966). ## [v1.87.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.3) Released at 2023-03-12 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See [the list of issues addressed in Go1.20.2](https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved). * BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3946). * BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692). -* BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during [background merge](./README.md#storage). The issue has been introduced at [v1.85.0](./CHANGELOG.md#v1850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897). +* BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during [background merge](https://docs.victoriametrics.com/#storage). The issue has been introduced at [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897). * BUGFIX: prevent from possible `SIGBUS` crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for [the initial attempt to fix it](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](./CHANGELOG.md#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). -* BUGFIX: [vmagent](./vmagent.md): do not register `vm_promscrape_config_*` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. -* BUGFIX: [vmctl](./vmctl.md): skip measurements with no fields when migrating data from influxdb. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3837). -* BUGFIX: [vmauth](./vmauth.md): fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt)-encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at [v1.87.0](./CHANGELOG.md#v1870) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since [v1.78.0](https://docs.victoriametrics.com/changelog/#v1780). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3932). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not register `vm_promscrape_config_*` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. +* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): skip measurements with no fields when migrating data from influxdb. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3837). +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt)-encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). ## [v1.87.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.2) Released at 2023-02-24 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** * SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See [alpine 3.17.2 release notes](https://alpinelinux.org/posts/Alpine-3.17.2-released.html). * SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See [the list of issues addressed in Go1.20.1](https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved). -* BUGFIX: [vmagent](./vmagent.md): immediately cancel in-flight scrape requests during configuration reload when [stream parsing mode](./vmagent.md#stream-parsing-mode) is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). -* BUGFIX: [vmagent](./vmagent.md): do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes [http keep-alive connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection) between scrapes. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747) issues. -* BUGFIX: [vmagent](./vmagent.md): fix [Azure service discovery](./sd_configs.md#azure_sd_configs) inside [Azure Container App](https://learn.microsoft.com/en-us/azure/container-apps/overview). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830). Thanks to @MattiasAng for the fix! +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): immediately cancel in-flight scrape requests during configuration reload when [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes [http keep-alive connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection) between scrapes. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747) issues. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix [Azure service discovery](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs) inside [Azure Container App](https://learn.microsoft.com/en-us/azure/container-apps/overview). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830). Thanks to @MattiasAng for the fix! * BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from `cannot create hard links from ...must-remove...` errors when making snapshots / backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3858). -* BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during [background merges of big parts](./README.md#storage) on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in [v1.85.0](./CHANGELOG.md#v1850) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790). -* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). -* BUGFIX: [MetricsQL](./MetricsQL.md): do not add extrapolated points outside the real points when using [interpolate()](./MetricsQL.md#interpolate) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3816). +* BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during [background merges of big parts](https://docs.victoriametrics.com/#storage) on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790). +* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): do not add extrapolated points outside the real points when using [interpolate()](https://docs.victoriametrics.com/metricsql/#interpolate) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3816). ## [v1.87.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.1) Released at 2023-02-09 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** -* FEATURE: [vmalert](./vmalert.md): alerts state restore procedure was changed to become asynchronous. It doesn't block groups start anymore which significantly improves vmalert's startup time. +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): alerts state restore procedure was changed to become asynchronous. It doesn't block groups start anymore which significantly improves vmalert's startup time. This also means that `-remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now and will have no effect if configured. While previously state restore attempt was made for all the loaded alerting rules, now it is called only for alerts which became active after the first evaluation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2608). -* FEATURE: [vmui](./README.md#vmui): optimize VMUI for use from smartphones and tablets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707). -* FEATURE: [vmui](./README.md#vmui): add ability to search tenants in the drop-down list for the tenant selector. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792). -* FEATURE: [vmui](./README.md#vmui): add avg/min/max/last values to line legends and tooltips for graphs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706). -* FEATURE: [vmui](./README.md#vmui): hide the default `per-job resource usage` dashboard if there is a custom dashboard exists at the directory specified via `-vmui.customDashboardsPath` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3740). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): optimize VMUI for use from smartphones and tablets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to search tenants in the drop-down list for the tenant selector. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add avg/min/max/last values to line legends and tooltips for graphs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): hide the default `per-job resource usage` dashboard if there is a custom dashboard exists at the directory specified via `-vmui.customDashboardsPath` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3740). -* BUGFIX: [vmagent](./vmagent.md): fix panic in [HashiCorp Nomad service discovery](./sd_configs.md#nomad_sd_configs). Thanks to @mr-karan for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3784). -* BUGFIX: [vmalert](./vmalert.md): fix display of rules number per-group for groups with identical names in UI. -* BUGFIX: [vmalert](./vmalert.md): prevent disabling state updates tracking per rule via setting values < 1. The minimum number of update states to track is now set to 1. -* BUGFIX: [vmalert](./vmalert.md): properly update `debug` and `update_entries_limit` rule's params on config's hot-reload. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix panic in [HashiCorp Nomad service discovery](https://docs.victoriametrics.com/sd_configs/#nomad_sd_configs). Thanks to @mr-karan for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3784). +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix display of rules number per-group for groups with identical names in UI. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): prevent disabling state updates tracking per rule via setting values < 1. The minimum number of update states to track is now set to 1. +* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly update `debug` and `update_entries_limit` rule's params on config's hot-reload. * BUGFIX: properly initialize the `vm_concurrent_insert_current` metric before exposing it. Previously this metric could be left uninitialized in some cases, e.g. its value was zero. This could lead to false alerts for the query `avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761). -* BUGFIX: [vmagent](./vmagent.md): immediately cancel in-flight scrape requests during configuration reload when using [stream parsing mode](./vmagent.md#stream-parsing-mode). Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). -* BUGFIX: [vmgateway](./vmgateway.md): do not validate JWT signature if no public keys are provided. Previously this could result in the `error setting up jwt verification` error. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): immediately cancel in-flight scrape requests during configuration reload when using [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747). +* BUGFIX: [vmgateway](https://docs.victoriametrics.com/vmgateway/): do not validate JWT signature if no public keys are provided. Previously this could result in the `error setting up jwt verification` error. ## [v1.87.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.0) Released at 2023-02-01 -**v1.87.x is a line of [LTS releases](./LTS-releases.md). It contains important up-to-date bugfixes. -The v1.87.x line will be supported for at least 12 months since [v1.87.0](./CHANGELOG.md#v1870) release** +**v1.87.x is a line of [LTS releases](https://docs.victoriametrics.com/lts-releases/). It contains important up-to-date bugfixes. +The v1.87.x line will be supported for at least 12 months since [v1.87.0](https://docs.victoriametrics.com/changelog/#v1870) release** -* FEATURE: [stream aggregation](./stream-aggregation.md): add the ability to [de-duplicate](./README.md#deduplication) input samples before aggregation via `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line options. -* FEATURE: [vmui](./README.md#vmui): add dark mode - it can be selected via `settings` menu in the top right corner. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3704). -* FEATURE: [vmui](./README.md#vmui): improve visual appearance of the top menu. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3678). -* FEATURE: [vmui](./README.md#vmui): embed fonts into binary instead of loading them from external sources. This allows using `vmui` in full from isolated networks without access to Internet. Thanks to @ScottKevill for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3696). -* FEATURE: [vmui](./README.md#vmui): add ability to switch between tenants by selecting the needed tenant in the drop-down list at the top right corner of the UI. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3673). -* FEATURE: [vmagent](./vmagent.md): reduce memory usage when sending stale markers for targets, which expose big number of metrics. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675) issues. -* FEATURE: [vmagent](./vmagent.md): add `__meta_kubernetes_pod_container_id` meta-label to the targets discovered via [kubernetes_sd_configs](./sd_configs.md#kubernetes_sd_configs). This label has been added in Prometheus starting from `v2.42.0`. See [this feature request](https://github.com/prometheus/prometheus/issues/11843). -* FEATURE: [vmagent](./vmagent.md): add `__meta_azure_machine_size` meta-label to the targets discovered via [azure_sd_configs](./sd_configs.md#azure_sd_configs). This label has been added in Prometheus starting from `v2.42.0`. See [this pull request](https://github.com/prometheus/prometheus/pull/11650). -* FEATURE: [vmauth](./vmauth.md): allow limiting the number of concurrent requests sent to `vmauth` via `-maxConcurrentRequests` command-line flag. This allows controlling memory usage of `vmauth` and the resource usage of backends behind `vmauth`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346). Thanks to @dmitryk-dk for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486). -* FEATURE: allow using VictoriaMetrics components behind proxies, which communicate with the backend via [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). For example, [vmauth](./vmauth.md) accepts proxy protocol connections when it starts with `-httpListenAddr.useProxyProtocol` command-line flag. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add the ability to [de-duplicate](https://docs.victoriametrics.com/#deduplication) input samples before aggregation via `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line options. +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add dark mode - it can be selected via `settings` menu in the top right corner. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3704). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve visual appearance of the top menu. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3678). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): embed fonts into binary instead of loading them from external sources. This allows using `vmui` in full from isolated networks without access to Internet. Thanks to @ScottKevill for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3696). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to switch between tenants by selecting the needed tenant in the drop-down list at the top right corner of the UI. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3673). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce memory usage when sending stale markers for targets, which expose big number of metrics. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675) issues. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_kubernetes_pod_container_id` meta-label to the targets discovered via [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs). This label has been added in Prometheus starting from `v2.42.0`. See [this feature request](https://github.com/prometheus/prometheus/issues/11843). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `__meta_azure_machine_size` meta-label to the targets discovered via [azure_sd_configs](https://docs.victoriametrics.com/sd_configs/#azure_sd_configs). This label has been added in Prometheus starting from `v2.42.0`. See [this pull request](https://github.com/prometheus/prometheus/pull/11650). +* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow limiting the number of concurrent requests sent to `vmauth` via `-maxConcurrentRequests` command-line flag. This allows controlling memory usage of `vmauth` and the resource usage of backends behind `vmauth`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346). Thanks to @dmitryk-dk for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486). +* FEATURE: allow using VictoriaMetrics components behind proxies, which communicate with the backend via [proxy protocol](https://www.haproxy.org/download/2.3/doc/proxy-protocol.txt). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335). For example, [vmauth](https://docs.victoriametrics.com/vmauth/) accepts proxy protocol connections when it starts with `-httpListenAddr.useProxyProtocol` command-line flag. * FEATURE: add `-internStringMaxLen` command-line flag, which can be used for fine-tuning RAM vs CPU usage in certain workloads. For example, if the stored time series contain long labels, then it may be useful reducing the `-internStringMaxLen` in order to reduce memory usage at the cost of increased CPU usage. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692). * FEATURE: provide GOARCH=386 binaries for single-node VictoriaMetrics, vmagent, vmalert, vmauth, vmbackup and vmrestore components at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3661). Thanks to @denisgolius for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3725). * BUGFIX: fix a bug, which could prevent background merges for the previous partitions until restart if the storage didn't have enough disk space for final deduplication and down-sampling. -* BUGFIX: fix a bug, which could lead to increased CPU usage and disk IO usage when adding data to previous months and when the [deduplication](./README.md#deduplication) or [downsampling](./README.md#downsampling) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3737). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): propagate all the timeout-related errors from `vmstorage` to `vmselect`. Previously some timeout errors weren't returned from `vmselect` to `vmstorage`. Instead, `vmstorage` could log the error and close the connection to `vmselect`, so `vmselect` was logging cryptic errors such as `cannot execute funcName="..." on vmstorage "...": EOF`. -* BUGFIX: [vmui](./README.md#vmui): add support for time zone selection for older versions of browsers. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3680). -* BUGFIX: [vmagent](./vmagent.md): update API version for [ec2_sd_configs](./sd_configs.md#ec2_sd_configs) to fix [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700) with missing `__meta_ec2_availability_zone_id` attribute. -* BUGFIX: [vmagent](./vmagent.md): properly return `200 OK` HTTP status code when importing data via [Pushgateway protocol](./README.md#how-to-import-data-in-prometheus-exposition-format). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636). -* BUGFIX: [vmagent](./vmagent.md): do not add `exported_` prefix to scraped metric names, which clash with the [automatically generated metric names](./vmagent.md#automatically-generated-metrics) if `honor_labels: true` option is set in the [scrape_config](./sd_configs.md#scrape_configs). See the [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3557) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406) issues. -* BUGFIX: [vmauth](./vmauth.md): allow re-entering authorization info in the web browser if the entered info was incorrect. Previously it was non-trivial to do via the web browser, since `vmauth` was returning `400 Bad Request` instead of `401 Unauthorized` http response code. -* BUGFIX: [vmauth](./vmauth.md): always log the client address and the requested URL on proxying errors. Previously some errors could miss this information. -* BUGFIX: [vmbackup](./vmbackup.md): fix snapshot not being deleted after backup completion. This issue could result in unnecessary snapshots being stored, it is required to delete unnecessary snapshots manually. See the [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3735). -* BUGFIX: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): fix panic on top-level vmselect nodes of [multi-level setup](./Cluster-VictoriaMetrics.md#multi-level-cluster-setup) when the `-replicationFactor` flag is set and request contains `trace` query parameter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3734). +* BUGFIX: fix a bug, which could lead to increased CPU usage and disk IO usage when adding data to previous months and when the [deduplication](https://docs.victoriametrics.com/#deduplication) or [downsampling](https://docs.victoriametrics.com/#downsampling) is enabled. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3737). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): propagate all the timeout-related errors from `vmstorage` to `vmselect`. Previously some timeout errors weren't returned from `vmselect` to `vmstorage`. Instead, `vmstorage` could log the error and close the connection to `vmselect`, so `vmselect` was logging cryptic errors such as `cannot execute funcName="..." on vmstorage "...": EOF`. +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): add support for time zone selection for older versions of browsers. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3680). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): update API version for [ec2_sd_configs](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs) to fix [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700) with missing `__meta_ec2_availability_zone_id` attribute. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly return `200 OK` HTTP status code when importing data via [Pushgateway protocol](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not add `exported_` prefix to scraped metric names, which clash with the [automatically generated metric names](https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics) if `honor_labels: true` option is set in the [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs). See the [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3557) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406) issues. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): allow re-entering authorization info in the web browser if the entered info was incorrect. Previously it was non-trivial to do via the web browser, since `vmauth` was returning `400 Bad Request` instead of `401 Unauthorized` http response code. +* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth/): always log the client address and the requested URL on proxying errors. Previously some errors could miss this information. +* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): fix snapshot not being deleted after backup completion. This issue could result in unnecessary snapshots being stored, it is required to delete unnecessary snapshots manually. See the [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3735). +* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fix panic on top-level vmselect nodes of [multi-level setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) when the `-replicationFactor` flag is set and request contains `trace` query parameter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3734). ## [v1.86.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.2) Released at 2023-01-18 -* SECURITY: [vmbackup](./vmbackup.md): do not expose basic auth passwords from `-snapshot.createURL` and `-snapshot.deleteURL` command-line flags in logs. Thanks to @toanju for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3672). +* SECURITY: [vmbackup](https://docs.victoriametrics.com/vmbackup/): do not expose basic auth passwords from `-snapshot.createURL` and `-snapshot.deleteURL` command-line flags in logs. Thanks to @toanju for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3672). -* FEATURE: [vmui](./README.md#vmui): add ability to show custom dashboards at vmui by specifying a path to a directory with dashboard config files via `-vmui.customDashboardsPath` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322) and [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards). -* FEATURE: [vmui](./README.md#vmui): apply the `step` globally to all the displayed graphs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3574). -* FEATURE: [vmui](./README.md#vmui): improve the appearance of graph lines by using more visually distinct colors. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3571). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to show custom dashboards at vmui by specifying a path to a directory with dashboard config files via `-vmui.customDashboardsPath` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322) and [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): apply the `step` globally to all the displayed graphs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3574). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve the appearance of graph lines by using more visually distinct colors. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3571). -* BUGFIX: do not slow down concurrently executed queries during assisted merges, since assisted merges already prioritize data ingestion over queries. The probability of assisted merges has been increased starting from [v1.85.0](./CHANGELOG.md#v1850) because of internal refactoring. This could result in slowed down queries when there is a plenty of free CPU resources. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641) issues. +* BUGFIX: do not slow down concurrently executed queries during assisted merges, since assisted merges already prioritize data ingestion over queries. The probability of assisted merges has been increased starting from [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850) because of internal refactoring. This could result in slowed down queries when there is a plenty of free CPU resources. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641) issues. * BUGFIX: reduce the increased CPU usage at `vmselect` to v1.85.3 level when processing heavy queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641). -* BUGFIX: [retention filters](./README.md#retention-filters): fix `FATAL: cannot locate metric name for metricID=...: EOF` panic, which could occur when retention filters are enabled. -* BUGFIX: [vmagent](./vmagent.md): properly cancel in-flight service discovery requests for [consul_sd_configs](./sd_configs.md#consul_sd_configs) and [nomad_sd_configs](./sd_configs.md#nomad_sd_configs) when the service list changes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). -* BUGFIX: [vmagent](./vmagent.md): suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. -* BUGFIX: [vmagent](./vmagent.md): consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information. -* BUGFIX: [vmagent](./vmagent.md): do not send stale markers to remote storage for series exceeding the configured [series limit](./vmagent.md#cardinality-limiter). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660). -* BUGFIX: [vmagent](./vmagent.md): properly apply [series limit](./vmagent.md#cardinality-limiter) when [staleness tracking](./vmagent.md#prometheus-staleness-markers) is disabled. -* BUGFIX: [vmagent](./vmagent.md): reduce memory usage spikes when big number of scrape targets disappear at once. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668). Thanks to @lzfhust for [the initial fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3669). -* BUGFIX: [Pushgateway import](./README.md#how-to-import-data-in-prometheus-exposition-format): properly return `200 OK` HTTP response code. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly parse `M` and `Mi` suffixes as `1e6` multipliers in `1M` and `1Mi` numeric constants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664). The issue has been introduced in [v1.86.0](./CHANGELOG.md#v1860). -* BUGFIX: [vmui](./README.md#vmui): properly display range query results at `Table` view. For example, `up[5m]` query now shows all the raw samples for the last 5 minutes for the `up` metric at the `Table` view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3516). +* BUGFIX: [retention filters](https://docs.victoriametrics.com/#retention-filters): fix `FATAL: cannot locate metric name for metricID=...: EOF` panic, which could occur when retention filters are enabled. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly cancel in-flight service discovery requests for [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs) and [nomad_sd_configs](https://docs.victoriametrics.com/sd_configs/#nomad_sd_configs) when the service list changes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): do not send stale markers to remote storage for series exceeding the configured [series limit](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly apply [series limit](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) when [staleness tracking](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) is disabled. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce memory usage spikes when big number of scrape targets disappear at once. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668). Thanks to @lzfhust for [the initial fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3669). +* BUGFIX: [Pushgateway import](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format): properly return `200 OK` HTTP response code. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly parse `M` and `Mi` suffixes as `1e6` multipliers in `1M` and `1Mi` numeric constants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664). The issue has been introduced in [v1.86.0](https://docs.victoriametrics.com/changelog/#v1860). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly display range query results at `Table` view. For example, `up[5m]` query now shows all the raw samples for the last 5 minutes for the `up` metric at the `Table` view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3516). ## [v1.86.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.1) Released at 2023-01-10 -* BUGFIX: return correct query results over time series with gaps. The issue has been introduced in [v1.86.0](./CHANGELOG.md#v1860). -* BUGFIX: properly take into account the timeout passed by `vmselect` to `vmstorage` during query execution. This issue could result in the following error logs at `vmstorage` under load: `cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed`. The issue has been introduced in [v1.86.0](./CHANGELOG.md#v1860). +* BUGFIX: return correct query results over time series with gaps. The issue has been introduced in [v1.86.0](https://docs.victoriametrics.com/changelog/#v1860). +* BUGFIX: properly take into account the timeout passed by `vmselect` to `vmstorage` during query execution. This issue could result in the following error logs at `vmstorage` under load: `cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed`. The issue has been introduced in [v1.86.0](https://docs.victoriametrics.com/changelog/#v1860). ## [v1.86.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.0) Released at 2023-01-10 -**It is recommended upgrading to [VictoriaMetrics v1.86.1](./CHANGELOG.md#v1861) because v1.86.0 contains a bug, which could lead to incorrect query results over time series with gaps.** +**It is recommended upgrading to [VictoriaMetrics v1.86.1](https://docs.victoriametrics.com/changelog/#v1861) because v1.86.0 contains a bug, which could lead to incorrect query results over time series with gaps.** -**Update note 1:** This release changes the logic behind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections established from clients, which send data to VictoriaMetrics. Some of these connections could be temporarily idle. Such connections do not take significant CPU and memory resources, so there is no need in limiting their count. The new logic takes into account only those connections, which **actively** ingest new data to VictoriaMetrics and to [vmagent](./vmagent.md). This means that the default `-maxConcurrentInserts` value should handle cases, which could require increasing the value in the previous releases. So it is recommended trying to remove the explicitly set `-maxConcurrentInserts` command-line flag after upgrading to this release and verifying whether this reduces CPU and memory usage. +**Update note 1:** This release changes the logic behind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections established from clients, which send data to VictoriaMetrics. Some of these connections could be temporarily idle. Such connections do not take significant CPU and memory resources, so there is no need in limiting their count. The new logic takes into account only those connections, which **actively** ingest new data to VictoriaMetrics and to [vmagent](https://docs.victoriametrics.com/vmagent/). This means that the default `-maxConcurrentInserts` value should handle cases, which could require increasing the value in the previous releases. So it is recommended trying to remove the explicitly set `-maxConcurrentInserts` command-line flag after upgrading to this release and verifying whether this reduces CPU and memory usage. -**Update note 2:** The `vm_concurrent_addrows_current` and `vm_concurrent_addrows_capacity` metrics [exported](./Cluster-VictoriaMetrics.md#monitoring) by `vmstorage` are replaced with `vm_concurrent_insert_current` and `vm_concurrent_insert_capacity` metrics in order to be consistent with the corresponding metrics exported by `vminsert`. Please update queries in dahsboards and alerting rules with new metric names if old metric names are used there. +**Update note 2:** The `vm_concurrent_addrows_current` and `vm_concurrent_addrows_capacity` metrics [exported](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) by `vmstorage` are replaced with `vm_concurrent_insert_current` and `vm_concurrent_insert_capacity` metrics in order to be consistent with the corresponding metrics exported by `vminsert`. Please update queries in dahsboards and alerting rules with new metric names if old metric names are used there. -* FEATURE: [vmagent](./vmagent.md): add support for aggregation of incoming [samples](./keyConcepts.md#raw-samples) by time and by labels. See [these docs](./stream-aggregation.md) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460). -* FEATURE: [vmagent](./vmagent.md): reduce memory usage when scraping big number of targets without the need to enable [stream parsing mode](./vmagent.md#stream-parsing-mode). -* FEATURE: [vmagent](./vmagent.md): add support for Prometheus-compatible target discovery for [HashiCorp Nomad](https://www.nomadproject.io/) services via [nomad_sd_configs](./sd_configs.md#nomad_sd_configs). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367). Thanks to @mr-karan for [the implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3549). -* FEATURE: [vmagent](./vmagent.md): automatically pre-fetch `metric_relabel_configs` and the target labels when clicking on the `debug metrics relabeling` link at the `http://vmagent:8429/targets` page at the particular target. See [these docs](./vmagent.md#relabel-debug). -* FEATURE: [vmui](./README.md#vmui): add ability to explore metrics exported by a particular `job` / `instance`. See [these docs](./README.md#metrics-explorer) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386). -* FEATURE: allow passing partial `RFC3339` date/time to `time`, `start` and `end` query args at [querying APIs](./README.md#prometheus-querying-api-usage) and [export APIs](./README.md#how-to-export-time-series). For example, `2022` is equivalent to `2022-01-01T00:00:00Z`, while `2022-01-30T14` is equivalent to `2022-01-30T14:00:00Z`. See [these docs](./README.md#timestamp-formats). -* FEATURE: [MetricsQL](./MetricsQL.md): allow using unicode letters in identifiers. For example, `температура{город="Київ"}` is a valid MetricsQL expression now. Previously every non-ascii letters should be escaped with `\` char when used inside MetricsQL expression: `\т\е\м\п\е\р\а\т\у\р\а{\г\о\р\о\д="Киев"}`. Now both expressions are equivalent. Thanks to @hzwwww for the [pull request](https://github.com/VictoriaMetrics/metricsql/pull/7). -* FEATURE: [relabeling](./vmagent.md#relabeling): add support for `keepequal` and `dropequal` relabeling actions, which are supported by Prometheus starting from [v2.41.0](https://github.com/prometheus/prometheus/releases/tag/v2.41.0). These relabeling actions are almost identical to `keep_if_equal` and `drop_if_equal` relabeling actions supported by VictoriaMetrics since `v1.38.0` - see [these docs](./vmagent.md#relabeling-enhancements) - so it is recommended sticking to `keep_if_equal` and `drop_if_equal` actions instead of switching to `keepequal` and `dropequal`. -* FEATURE: [csvimport](./README.md#how-to-import-csv-data): support empty values for imported metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540). -* FEATURE: [vmalert](./vmalert.md): allow configuring the default number of stored rule's update states in memory via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's `update_entries_limit` configuration param. See [these docs](./vmalert.md#rules) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3556). -* FEATURE: improve the logic benhind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections from clients, which write data to VictoriaMetrics or [vmagent](./vmagent.md). Some of these connections could be idle for some time. These connections do not need significant amounts of CPU and memory, so there is no sense in limiting their count. The updated logic behind `-maxConcurrentInserts` limits the number of **active** insert requests, not counting idle connections. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for aggregation of incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) by time and by labels. See [these docs](https://docs.victoriametrics.com/stream-aggregation/) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce memory usage when scraping big number of targets without the need to enable [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for Prometheus-compatible target discovery for [HashiCorp Nomad](https://www.nomadproject.io/) services via [nomad_sd_configs](https://docs.victoriametrics.com/sd_configs/#nomad_sd_configs). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367). Thanks to @mr-karan for [the implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3549). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): automatically pre-fetch `metric_relabel_configs` and the target labels when clicking on the `debug metrics relabeling` link at the `http://vmagent:8429/targets` page at the particular target. See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug). +* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to explore metrics exported by a particular `job` / `instance`. See [these docs](https://docs.victoriametrics.com/#metrics-explorer) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386). +* FEATURE: allow passing partial `RFC3339` date/time to `time`, `start` and `end` query args at [querying APIs](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). For example, `2022` is equivalent to `2022-01-01T00:00:00Z`, while `2022-01-30T14` is equivalent to `2022-01-30T14:00:00Z`. See [these docs](https://docs.victoriametrics.com/#timestamp-formats). +* FEATURE: [MetricsQL](https://docs.victoriametrics.com/metricsql/): allow using unicode letters in identifiers. For example, `температура{город="Київ"}` is a valid MetricsQL expression now. Previously every non-ascii letters should be escaped with `\` char when used inside MetricsQL expression: `\т\е\м\п\е\р\а\т\у\р\а{\г\о\р\о\д="Киев"}`. Now both expressions are equivalent. Thanks to @hzwwww for the [pull request](https://github.com/VictoriaMetrics/metricsql/pull/7). +* FEATURE: [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling): add support for `keepequal` and `dropequal` relabeling actions, which are supported by Prometheus starting from [v2.41.0](https://github.com/prometheus/prometheus/releases/tag/v2.41.0). These relabeling actions are almost identical to `keep_if_equal` and `drop_if_equal` relabeling actions supported by VictoriaMetrics since `v1.38.0` - see [these docs](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements) - so it is recommended sticking to `keep_if_equal` and `drop_if_equal` actions instead of switching to `keepequal` and `dropequal`. +* FEATURE: [csvimport](https://docs.victoriametrics.com/#how-to-import-csv-data): support empty values for imported metrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): allow configuring the default number of stored rule's update states in memory via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's `update_entries_limit` configuration param. See [these docs](https://docs.victoriametrics.com/vmalert/#rules) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3556). +* FEATURE: improve the logic benhind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections from clients, which write data to VictoriaMetrics or [vmagent](https://docs.victoriametrics.com/vmagent/). Some of these connections could be idle for some time. These connections do not need significant amounts of CPU and memory, so there is no sense in limiting their count. The updated logic behind `-maxConcurrentInserts` limits the number of **active** insert requests, not counting idle connections. * FEATURE: protect all the http endpoints with `-httpAuth.*` command-line flag. Previously endpoints protected by `-*AuthKey` command-line flags weren't protected by `-httpAuth.*`. This could complicate the proper security setup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3060). -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): add `-maxConcurrentInserts` and `-insert.maxQueueDuration` command-line flags to `vmstorage`, so they could be tuned if needed in the same way as at `vminsert` nodes. -* FEATURE: [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md): limit the number of concurrently executed requests at `vmstorage` proportionally to the number of available CPU cores, since every request can saturate a single CPU core at `vmstorage`. Previously a single `vmstorage` could accept and start processing arbitrary number of concurrent requests received from big number of `vmselect` nodes. This could result in increased RAM, CPU and disk IO usage or event to out of memory crash at `vmstorage` side under high load. The limit can be fine-tuned if needed via `-search.maxConcurrentRequests` command-line flag at `vmstorage` according to [these docs](./Cluster-VictoriaMetrics.md#resource-usage-limits). `vmstorage` now [exposes](./Cluster-VictoriaMetrics.md#monitoring) the following additional metrics at `http://vmstorage:8482/metrics` page: +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): add `-maxConcurrentInserts` and `-insert.maxQueueDuration` command-line flags to `vmstorage`, so they could be tuned if needed in the same way as at `vminsert` nodes. +* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): limit the number of concurrently executed requests at `vmstorage` proportionally to the number of available CPU cores, since every request can saturate a single CPU core at `vmstorage`. Previously a single `vmstorage` could accept and start processing arbitrary number of concurrent requests received from big number of `vmselect` nodes. This could result in increased RAM, CPU and disk IO usage or event to out of memory crash at `vmstorage` side under high load. The limit can be fine-tuned if needed via `-search.maxConcurrentRequests` command-line flag at `vmstorage` according to [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits). `vmstorage` now [exposes](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) the following additional metrics at `http://vmstorage:8482/metrics` page: - `vm_vmselect_concurrent_requests_capacity` - the maximum number of requests allowed to execute concurrently - `vm_vmselect_concurrent_requests_current` - the current number of concurrently executed requests - `vm_vmselect_concurrent_requests_limit_reached_total` - the total number of requests, which were put in the wait queue when `-search.maxConcurrentRequests` concurrent requests are being executed - `vm_vmselect_concurrent_requests_limit_timeout_total` - the total number of canceled requests because they were sitting in the wait queue for more than `-search.maxQueueDuration` -* BUGFIX: [vmui](./README.md#vmui): properly update the `step` value in url after the `step` input field has been manually changed. This allows preserving the proper `step` when copy-n-pasting the url to another instance of web browser. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3513). -* BUGFIX: [vmui](./README.md#vmui): properly update tooltip when quickly hovering multiple lines on the graph. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3530). -* BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during [data import](./README.md#how-to-import-time-series-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544). -* BUGFIX: [MetricsQL](./MetricsQL.md): properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589). -* BUGFIX: [vmagent](./vmagent.md): fix a panic during target discovery when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580). The bug has been introduced in [v1.85.0](./CHANGELOG.md#v1850). -* BUGFIX: [vmagent](./vmagent.md): [dockerswarm_sd_configs](./sd_configs.md#dockerswarm_sd_configs): properly encode `filters` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). -* BUGFIX: [vmagent](./vmagent.md): fix possible resource leak after hot reload of the updated [consul_sd_configs](./sd_configs.md#consul_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468). -* BUGFIX: [vmagent](./vmagent.md): fix a panic in [gce_sd_configs](./sd_configs.md#gce_sd_configs) when the discovered instance has zero labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624). The issue has been introduced in [v1.85.0](./CHANGELOG.md#v1850). -* BUGFIX: properly return label names starting from uppercase such as `CamelCaseLabel` from [/api/v1/labels](./url-examples.md#apiv1labels). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3566). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly update the `step` value in url after the `step` input field has been manually changed. This allows preserving the proper `step` when copy-n-pasting the url to another instance of web browser. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3513). +* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly update tooltip when quickly hovering multiple lines on the graph. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3530). +* BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during [data import](https://docs.victoriametrics.com/#how-to-import-time-series-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544). +* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/metricsql/): properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix a panic during target discovery when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580). The bug has been introduced in [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs): properly encode `filters` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix possible resource leak after hot reload of the updated [consul_sd_configs](https://docs.victoriametrics.com/sd_configs/#consul_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): fix a panic in [gce_sd_configs](https://docs.victoriametrics.com/sd_configs/#gce_sd_configs) when the discovered instance has zero labels. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624). The issue has been introduced in [v1.85.0](https://docs.victoriametrics.com/changelog/#v1850). +* BUGFIX: properly return label names starting from uppercase such as `CamelCaseLabel` from [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3566). * BUGFIX: fix `opentsdb` HTTP endpoint not respecting `-httpAuth.*` flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3060) -* BUGFIX: consistently select the sample with the biggest value out of samples with identical timestamps during querying when the [deduplication](./README.md#deduplication) is enabled according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333). Previously random samples could be selected during querying. +* BUGFIX: consistently select the sample with the biggest value out of samples with identical timestamps during querying when the [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333). Previously random samples could be selected during querying. ## Previous releases -See changes for older releases [here](./CHANGELOG_2022.md). +See changes for older releases [here](https://docs.victoriametrics.com/changelog_2022/). diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md index 193bf75f4..b294d6dda 100644 --- a/docs/Cluster-VictoriaMetrics.md +++ b/docs/Cluster-VictoriaMetrics.md @@ -22,20 +22,20 @@ for ingestion rates lower than a million data points per second. The single-node version [scales perfectly](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae) with the number of CPU cores, RAM and available storage space. The single-node version is easier to configure and operate compared to the cluster version, so think twice before choosing the cluster version. -See [this question](./FAQ.md#which-victoriametrics-type-is-recommended-for-use-in-production---single-node-or-cluster) for more details. +See [this question](https://docs.victoriametrics.com/faq/#which-victoriametrics-type-is-recommended-for-use-in-production---single-node-or-cluster) for more details. -There is also user-friendly database for logs - [VictoriaLogs](./VictoriaLogs/README.md). +There is also user-friendly database for logs - [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/). If you have questions about VictoriaMetrics, then feel free asking them at [VictoriaMetrics community Slack chat](https://victoriametrics.slack.com/), you can join it via [Slack Inviter](https://slack.victoriametrics.com/). [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. -See [features available in enterprise package](./enterprise.md). +See [features available in enterprise package](https://docs.victoriametrics.com/enterprise/). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/). -VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](./CHANGELOG.md) and performing [regular upgrades](#how-to-upgrade-victoriametrics). +VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/changelog/) and performing [regular upgrades](#how-to-upgrade-victoriametrics). VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details. @@ -72,8 +72,8 @@ Some facts about tenants in VictoriaMetrics: - Each `accountID` and `projectID` is identified by an arbitrary 32-bit integer in the range `[0 .. 2^32)`. If `projectID` is missing, then it is automatically assigned to `0`. It is expected that other information about tenants such as auth tokens, tenant names, limits, accounting, etc. is stored in a separate relational database. This database must be managed -by a separate service sitting in front of VictoriaMetrics cluster such as [vmauth](./vmauth.md) -or [vmgateway](./vmgateway.md). [Contact us](mailto:info@victoriametrics.com) if you need assistance with such service. +by a separate service sitting in front of VictoriaMetrics cluster such as [vmauth](https://docs.victoriametrics.com/vmauth/) +or [vmgateway](https://docs.victoriametrics.com/vmgateway/). [Contact us](mailto:info@victoriametrics.com) if you need assistance with such service. - Tenants are automatically created when the first data point is written into the given tenant. @@ -86,7 +86,7 @@ when different tenants have different amounts of data and different query load. - The list of registered tenants can be obtained via `http://:8481/admin/tenants` url. See [these docs](#url-format). -- VictoriaMetrics exposes various per-tenant statistics via metrics - see [these docs](./PerTenantStatistic.md). +- VictoriaMetrics exposes various per-tenant statistics via metrics - see [these docs](https://docs.victoriametrics.com/pertenantstatistic/). See also [multitenancy via labels](#multitenancy-via-labels). @@ -108,13 +108,13 @@ http_requests_total{path="/bar",vm_account_id="7",vm_project_id="9"} 34 Then the `http_requests_total{path="/foo"} 12` would be stored in the tenant `accountID=42, projectID=0`, while the `http_requests_total{path="/bar"} 34` would be stored in the tenant `accountID=7, projectID=9`. -The `vm_account_id` and `vm_project_id` labels are extracted after applying the [relabeling](./relabeling.md) +The `vm_account_id` and `vm_project_id` labels are extracted after applying the [relabeling](https://docs.victoriametrics.com/relabeling/) set via `-relabelConfig` command-line flag, so these labels can be set at this stage. The `vm_account_id` and `vm_project_id` labels are also taken into account when ingesting data via non-http-based protocols -such as [Graphite](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd), -[InfluxDB line protocol via TCP and UDP](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and -[OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol). +such as [Graphite](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd), +[InfluxDB line protocol via TCP and UDP](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and +[OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). **Security considerations:** it is recommended restricting access to `multitenant` endpoints only to trusted sources, since untrusted source may break per-tenant data by writing unwanted samples to arbitrary tenants. @@ -191,14 +191,14 @@ A minimal cluster must contain the following nodes: - a single `vminsert` node with `-storageNode=` - a single `vmselect` node with `-storageNode=` -[Enterprise version of VictoriaMetrics](./enterprise.md) supports automatic discovering and updating of `vmstorage` nodes. +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports automatic discovering and updating of `vmstorage` nodes. See [these docs](#automatic-vmstorage-discovery) for details. It is recommended to run at least two nodes for each service for high availability purposes. In this case the cluster continues working when a single node is temporarily unavailable and the remaining nodes can handle the increased workload. The node may be temporarily unavailable when the underlying hardware breaks, during software upgrades, migration or other maintenance tasks. It is preferred to run many small `vmstorage` nodes over a few big `vmstorage` nodes, since this reduces the workload increase on the remaining `vmstorage` nodes when some of `vmstorage` nodes become temporarily unavailable. -An http load balancer such as [vmauth](./vmauth.md) or `nginx` must be put in front of `vminsert` and `vmselect` nodes. +An http load balancer such as [vmauth](https://docs.victoriametrics.com/vmauth/) or `nginx` must be put in front of `vminsert` and `vmselect` nodes. It must contain the following routing configs according to [the url format](#url-format): - requests starting with `/insert` must be routed to port `8480` on `vminsert` nodes. @@ -237,7 +237,7 @@ Additionally, all the VictoriaMetrics components allow setting flag values via e ## Automatic vmstorage discovery -`vminsert` and `vmselect` components in [enterprise version of VictoriaMetrics](./enterprise.md) support +`vminsert` and `vmselect` components in [enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) support the following approaches for automatic discovery of `vmstorage` nodes: - file-based discovery - put the list of `vmstorage` nodes into a file - one node address per each line - and then pass `-storageNode=file:/path/to/file-with-vmstorage-list` @@ -271,23 +271,23 @@ The currently discovered `vmstorage` nodes can be [monitored](#monitoring) with General security recommendations: - All the VictoriaMetrics cluster components must run in protected private network without direct access from untrusted networks such as Internet. -- External clients must access `vminsert` and `vmselect` via auth proxy such as [vmauth](./vmauth.md) - or [vmgateway](./vmgateway.md). +- External clients must access `vminsert` and `vmselect` via auth proxy such as [vmauth](https://docs.victoriametrics.com/vmauth/) + or [vmgateway](https://docs.victoriametrics.com/vmgateway/). - The auth proxy must accept auth tokens from untrusted networks only via https in order to protect the auth tokens from MitM attacks. - It is recommended using distinct auth tokens for distinct [tenants](#multitenancy) in order to reduce potential damage in case of compromised auth token for some tenants. - Prefer using lists of allowed [API endpoints](#url-format), while disallowing access to other endpoints when configuring auth proxy in front of `vminsert` and `vmselect`. This minimizes attack surface. -See also [security recommendation for single-node VictoriaMetrics](./README.md#security) +See also [security recommendation for single-node VictoriaMetrics](https://docs.victoriametrics.com/#security) and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/). ## mTLS protection By default `vminsert` and `vmselect` nodes accept http requests at `8480` and `8481` ports accordingly (these ports can be changed via `-httpListenAddr` command-line flags), -since it is expected that [vmauth](./vmauth.md) is used for authorization and [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy) +since it is expected that [vmauth](https://docs.victoriametrics.com/vmauth/) is used for authorization and [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy) in front of `vminsert` and `vmselect`. -[Enterprise version of VictoriaMetrics](./enterprise.md) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) requests at `8480` and `8481` ports for `vminsert` and `vmselect` nodes, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `vmselect`, which accepts only mTLS requests at port `8481`: @@ -298,7 +298,7 @@ For example, the following command runs `vmselect`, which accepts only mTLS requ By default system-wide [TLS Root CA](https://en.wikipedia.org/wiki/Root_certificate) is used for verifying client certificates if `-mtls` command-line flag is specified. It is possible to specify custom TLS Root CA via `-mtlsCAFile` command-line flag. -By default `vminsert` and `vmselect` nodes use unencrypted connections to `vmstorage` nodes, since it is assumed that all the cluster components [run in a protected environment](#security). [Enterprise version of VictoriaMetrics](./enterprise.md) provides optional support for [mTLS connections](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) between cluster components. Pass `-cluster.tls=true` command-line flag to `vminsert`, `vmselect` and `vmstorage` nodes in order to enable mTLS protection. Additionally, `vminsert`, `vmselect` and `vmstorage` must be configured with mTLS certificates via `-cluster.tlsCertFile`, `-cluster.tlsKeyFile` command-line options. These certificates are mutually verified when `vminsert` and `vmselect` dial `vmstorage`. +By default `vminsert` and `vmselect` nodes use unencrypted connections to `vmstorage` nodes, since it is assumed that all the cluster components [run in a protected environment](#security). [Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) provides optional support for [mTLS connections](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) between cluster components. Pass `-cluster.tls=true` command-line flag to `vminsert`, `vmselect` and `vmstorage` nodes in order to enable mTLS protection. Additionally, `vminsert`, `vmselect` and `vmstorage` must be configured with mTLS certificates via `-cluster.tlsCertFile`, `-cluster.tlsKeyFile` command-line options. These certificates are mutually verified when `vminsert` and `vmselect` dial `vmstorage`. The following optional command-line flags related to mTLS are supported: @@ -310,7 +310,7 @@ When `vmselect` runs with `-clusternativeListenAddr` command-line option, then i See [these docs](https://gist.github.com/f41gh7/76ed8e5fb1ebb9737fe746bae9175ee6) on how to set up mTLS in VictoriaMetrics cluster. -[Enterprise version of VictoriaMetrics](./enterprise.md) can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/). ## Monitoring @@ -322,7 +322,7 @@ By default, the following TCP ports are used: - `vmselect` - 8481 - `vmstorage` - 8482 -It is recommended setting up [vmagent](./vmagent.md) +It is recommended setting up [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus to scrape `/metrics` pages from all the cluster components, so they can be monitored and analyzed with [the official Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176) or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11831). @@ -332,27 +332,27 @@ If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMet command-line to them, so they add `TYPE` and `HELP` comments per each exposed metric at `/metrics` page. See [these docs](https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type) for details. -It is recommended setting up alerts in [vmalert](./vmalert.md) or in Prometheus from [this list](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts). +It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert/) or in Prometheus from [this list](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts). See more details in the article [VictoriaMetrics Monitoring](https://victoriametrics.com/blog/victoriametrics-monitoring/). ## Cardinality limiter `vmstorage` nodes can be configured with limits on the number of unique time series across all the tenants with the following command-line flags: -- `-storage.maxHourlySeries` is the limit on the number of [active time series](./FAQ.md#what-is-an-active-time-series) during the last hour. -- `-storage.maxDailySeries` is the limit on the number of unique time series during the day. This limit can be used for limiting daily [time series churn rate](./FAQ.md#what-is-high-churn-rate). +- `-storage.maxHourlySeries` is the limit on the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) during the last hour. +- `-storage.maxDailySeries` is the limit on the number of unique time series during the day. This limit can be used for limiting daily [time series churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). Note that these limits are set and applied individually per each `vmstorage` node in the cluster. So, if the cluster has `N` `vmstorage` nodes, then the cluster-level limits will be `N` times bigger than the per-`vmstorage` limits. -See more details about cardinality limiter in [these docs](./README.md#cardinality-limiter). +See more details about cardinality limiter in [these docs](https://docs.victoriametrics.com/#cardinality-limiter). ## Troubleshooting - If your VictoriaMetrics cluster experiences data ingestion delays during - [rolling restarts and configuration updates](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes), + [rolling restarts and configuration updates](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes), then see [these docs](#improving-re-routing-performance-during-restart). -[Troubleshooting docs for single-node VictoriaMetrics](./Troubleshooting.md) apply to VictoriaMetrics cluster as well. +[Troubleshooting docs for single-node VictoriaMetrics](https://docs.victoriametrics.com/troubleshooting/) apply to VictoriaMetrics cluster as well. ## Readonly mode @@ -366,10 +366,10 @@ The metric is set to `0` when the `vmstorage` isn't in read-only mode. ## URL format -The main differences between URL formats of cluster and [Single server](./Single-Server-VictoriaMetrics.md) +The main differences between URL formats of cluster and [Single server](https://docs.victoriametrics.com/single-server-victoriametrics/) versions are that cluster has separate components for read and ingestion path, and because of multi-tenancy support. Also in the cluster version the `/prometheus/api/v1` endpoint ingests `jsonl`, `csv`, `native` and `prometheus` data formats **not** only `prometheus` data. -Check practical examples of VictoriaMetrics API [here](./url-examples.md). +Check practical examples of VictoriaMetrics API [here](https://docs.victoriametrics.com/url-examples/). - URLs for data ingestion: `http://:8480/insert//`, where: - `` is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant). It is possible to set it as `accountID:projectID`, @@ -380,34 +380,34 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md). - `prometheus` and `prometheus/api/v1/write` - for ingesting data with [Prometheus remote write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). - `prometheus/api/v1/import` - for importing data obtained via `api/v1/export` at `vmselect` (see below), JSON line format. - `prometheus/api/v1/import/native` - for importing data obtained via `api/v1/export/native` on `vmselect` (see below). - - `prometheus/api/v1/import/csv` - for importing arbitrary CSV data. See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-csv-data) for details. - - `prometheus/api/v1/import/prometheus` - for importing data in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md). This endpoint also supports [Pushgateway protocol](https://github.com/prometheus/pushgateway#url). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-data-in-prometheus-exposition-format) for details. - - `opentelemetry/v1/metrics` - for ingesting data via [OpenTelemetry protocol for metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/ffddc289462dfe0c2041e3ca42a7b1df805706de/specification/metrics/data-model.md). See [these docs](./Single-Server-VictoriaMetrics.md#sending-data-via-opentelemetry). - - `datadog/api/v1/series` - for ingesting data with DataDog submit metrics API v1. See [these docs](./url-examples.md#datadogapiv1series) for details. - - `datadog/api/v2/series` - for ingesting data with [DataDog submit metrics API](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent) for details. + - `prometheus/api/v1/import/csv` - for importing arbitrary CSV data. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-csv-data) for details. + - `prometheus/api/v1/import/prometheus` - for importing data in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md). This endpoint also supports [Pushgateway protocol](https://github.com/prometheus/pushgateway#url). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-data-in-prometheus-exposition-format) for details. + - `opentelemetry/v1/metrics` - for ingesting data via [OpenTelemetry protocol for metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/ffddc289462dfe0c2041e3ca42a7b1df805706de/specification/metrics/data-model.md). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#sending-data-via-opentelemetry). + - `datadog/api/v1/series` - for ingesting data with DataDog submit metrics API v1. See [these docs](https://docs.victoriametrics.com/url-examples/#datadogapiv1series) for details. + - `datadog/api/v2/series` - for ingesting data with [DataDog submit metrics API](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-datadog-agent) for details. - `datadog/api/beta/sketches` - for ingesting data with [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/). - - `influx/write` and `influx/api/v2/write` - for ingesting data with [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/). TCP and UDP receiver is disabled by default. It is exposed on a distinct TCP address set via `-influxListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) for details. - - `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./README.md#how-to-send-data-from-newrelic-agent) for details. - - `opentsdb/api/put` - for accepting [OpenTSDB HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html). This handler is disabled by default. It is exposed on a distinct TCP address set via `-opentsdbHTTPListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#sending-opentsdb-data-via-http-apiput-requests) for details. + - `influx/write` and `influx/api/v2/write` - for ingesting data with [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/). TCP and UDP receiver is disabled by default. It is exposed on a distinct TCP address set via `-influxListenAddr` command-line flag. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) for details. + - `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](https://docs.victoriametrics.com/#how-to-send-data-from-newrelic-agent) for details. + - `opentsdb/api/put` - for accepting [OpenTSDB HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html). This handler is disabled by default. It is exposed on a distinct TCP address set via `-opentsdbHTTPListenAddr` command-line flag. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#sending-opentsdb-data-via-http-apiput-requests) for details. - URLs for [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): `http://:8481/select//prometheus/`, where: - `` is an arbitrary number identifying data namespace for the query (aka tenant) - `` may have the following values: - - `api/v1/query` - performs [PromQL instant query](./keyConcepts.md#instant-query). - - `api/v1/query_range` - performs [PromQL range query](./keyConcepts.md#range-query). - - `api/v1/series` - performs [series query](./url-examples.md#apiv1series). - - `api/v1/labels` - returns a [list of label names](./url-examples.md#apiv1labels). - - `api/v1/label//values` - returns values for the given `` according [to the API](./url-examples.md#apiv1labelvalues). + - `api/v1/query` - performs [PromQL instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query). + - `api/v1/query_range` - performs [PromQL range query](https://docs.victoriametrics.com/keyconcepts/#range-query). + - `api/v1/series` - performs [series query](https://docs.victoriametrics.com/url-examples/#apiv1series). + - `api/v1/labels` - returns a [list of label names](https://docs.victoriametrics.com/url-examples/#apiv1labels). + - `api/v1/label//values` - returns values for the given `` according [to the API](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). - `federate` - returns [federated metrics](https://prometheus.io/docs/prometheus/latest/federation/). - `api/v1/export` - exports raw data in JSON line format. See [this article](https://medium.com/@valyala/analyzing-prometheus-data-with-external-tools-5f3e5e147639) for details. - `api/v1/export/native` - exports raw data in native binary format. It may be imported into another VictoriaMetrics via `api/v1/import/native` (see above). - `api/v1/export/csv` - exports data in CSV. It may be imported into another VictoriaMetrics via `api/v1/import/csv` (see above). - `api/v1/series/count` - returns the total number of series. - - `api/v1/status/tsdb` - for time series stats. See [these docs](./README.md#tsdb-stats) for details. + - `api/v1/status/tsdb` - for time series stats. See [these docs](https://docs.victoriametrics.com/#tsdb-stats) for details. - `api/v1/status/active_queries` - for currently executed active queries. Note that every `vmselect` maintains an independent list of active queries, which is returned in the response. - `api/v1/status/top_queries` - for listing the most frequently executed queries and queries taking the most duration. - - `metric-relabel-debug` - for debugging [relabeling rules](./relabeling.md). + - `metric-relabel-debug` - for debugging [relabeling rules](https://docs.victoriametrics.com/relabeling/). - URLs for [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api): `http://:8481/select//graphite/`, where: - `` is an arbitrary number identifying data namespace for query (aka tenant) @@ -436,11 +436,11 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md). - URL for listing [tenants](#multitenancy) with the ingested data on the given time range: `http://:8481/admin/tenants?start=...&end=...` . The `start` and `end` query args are optional. If they are missing, then all the tenants with at least one sample stored in VictoriaMetrics are returned. -- URL for accessing [vmalerts](./vmalert.md) UI: `http://:8481/select//prometheus/vmalert/`. +- URL for accessing [vmalerts](https://docs.victoriametrics.com/vmalert/) UI: `http://:8481/select//prometheus/vmalert/`. This URL works only when `-vmalert.proxyURL` flag is set. See more about vmalert [here](#vmalert). - `vmstorage` nodes provide the following HTTP endpoints on `8482` port: - - `/internal/force_merge` - initiate [forced compactions](./README.md#forced-merge) on the given `vmstorage` node. + - `/internal/force_merge` - initiate [forced compactions](https://docs.victoriametrics.com/#forced-merge) on the given `vmstorage` node. - `/snapshot/create` - create [instant snapshot](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282), which can be used for backups in background. Snapshots are created in `/snapshots` folder, where `` is the corresponding command-line flag value. @@ -463,8 +463,8 @@ General recommendations for cluster scalability: - Adding more CPU and RAM to existing `vmselect` nodes improves the performance for heavy queries, which process big number of time series with big number of raw samples. See [this article on how to detect and optimize heavy queries](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986). -- Adding more `vmstorage` nodes (aka horizontal scaling) increases the number of [active time series](./FAQ.md#what-is-an-active-time-series) - the cluster can handle. This also increases query performance over time series with [high churn rate](./FAQ.md#what-is-high-churn-rate), +- Adding more `vmstorage` nodes (aka horizontal scaling) increases the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) + the cluster can handle. This also increases query performance over time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), since every `vmstorage` node contains lower number of time series when the number of `vmstorage` nodes increases. The cluster stability is also improved with the number of `vmstorage` nodes, since active `vmstorage` nodes need to handle lower additional workload @@ -473,9 +473,9 @@ General recommendations for cluster scalability: If one node out of 10 nodes is unavailable, then `1/10=10%` of the load is re-distributed across 9 remaining nodes, so per-node workload increase is `(1/10/9)/(1/10) = 1/9 =~ 11%`. - Adding more CPU and RAM to existing `vmstorage` nodes (aka vertical scaling) increases the number - of [active time series](./FAQ.md#what-is-an-active-time-series) the cluster can handle. + of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) the cluster can handle. It is preferred to add more `vmstorage` nodes over adding more CPU and RAM to existing `vmstorage` nodes, since higher number of `vmstorage` nodes - increases cluster stability and improves query performance over time series with [high churn rate](./FAQ.md#what-is-high-churn-rate). + increases cluster stability and improves query performance over time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). - Adding more `vminsert` nodes increases the maximum possible data ingestion speed, since the ingested data may be split among bigger number of `vminsert` nodes. @@ -516,7 +516,7 @@ This strategy allows upgrading the cluster without downtime if the following con the current workload when a single node of any type (`vminsert`, `vmselect` or `vmstorage`) is temporarily unavailable during its restart. - The updated config / upgraded binary is compatible with the remaining components in the cluster. - See the [CHANGELOG](./CHANGELOG.md) for compatibility notes between different releases. + See the [CHANGELOG](https://docs.victoriametrics.com/changelog/) for compatibility notes between different releases. If at least a single condition isn't met, then the rolling restart may result in cluster unavailability during the config update / version upgrade. In this case the following strategy is recommended. @@ -542,7 +542,7 @@ The `minimum downtime` strategy has the following benefits comparing to `no down ## Improving re-routing performance during restart `vmstorage` nodes may experience increased usage for CPU, RAM and disk IO during -[rolling restarts](./Cluster-VictoriaMetrics.md#no-downtime-strategy), +[rolling restarts](https://docs.victoriametrics.com/cluster-victoriametrics/#no-downtime-strategy), since they need to process higher load when some of `vmstorage` nodes are temporarily unavailable in the cluster. The following approaches can be used for reducing resource usage at `vmstorage` nodes during rolling restart: @@ -552,7 +552,7 @@ The following approaches can be used for reducing resource usage at `vmstorage` Note that the `-disableReroutingOnUnavailable` flag may pause data ingestion for long time when some `vmstorage` nodes are unavailable for long time. -- To pass bigger values to `-storage.vminsertConnsShutdownDuration` (available from [v1.95.0](./CHANGELOG.md#v1950)) +- To pass bigger values to `-storage.vminsertConnsShutdownDuration` (available from [v1.95.0](https://docs.victoriametrics.com/changelog/#v1950)) command-line flag at `vmstorage` nodes.In this case `vmstorage` increases the interval between gradual closing of `vminsert` connections during graceful shutdown. This reduces data ingestion slowdown during rollout restarts. @@ -571,7 +571,7 @@ if some of its components are temporarily unavailable. VictoriaMetrics cluster remains available if the following conditions are met: - HTTP load balancer must stop routing requests to unavailable `vminsert` and `vmselect` nodes - ([vmauth](./vmauth.md) stops routing requests to unavailable nodes). + ([vmauth](https://docs.victoriametrics.com/vmauth/) stops routing requests to unavailable nodes). - At least a single `vminsert` node must remain available in the cluster for processing data ingestion workload. The remaining active `vminsert` nodes must have enough compute capacity (CPU, RAM, network bandwidth) @@ -601,7 +601,7 @@ The cluster works in the following way when some of `vmstorage` nodes are unavai for processing the increased data ingestion workload. `vminsert` spreads evenly the additional data among the healthy `vmstorage` nodes in order to spread evenly the increased load on these nodes. During re-routing, healthy `vmstorage` nodes will experience higher resource usage - and increase in number of [active time series](./FAQ.md#what-is-an-active-time-series). + and increase in number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). - `vmselect` continues serving queries if at least a single `vmstorage` nodes is available. It marks responses as partial for queries served from the remaining healthy `vmstorage` nodes, @@ -617,13 +617,13 @@ The cluster works in the following way when some of `vmstorage` nodes are unavai It is also possible to configure independent replication factor per distinct `vmstorage` groups - see [these docs](#vmstorage-groups-at-vmselect). -`vmselect` doesn't serve partial responses for API handlers returning [raw datapoints](./keyConcepts.md#raw-samples), +`vmselect` doesn't serve partial responses for API handlers returning [raw datapoints](https://docs.victoriametrics.com/keyconcepts/#raw-samples), since users usually expect this data is always complete. The following handlers return raw samples: -- [`/api/v1/export*` endpoints](./README.md#how-to-export-time-series) -- [`/api/v1/query`](./url-examples.md#apiv1query) when the `query` contains [series selector](./keyConcepts.md#filtering) +- [`/api/v1/export*` endpoints](https://docs.victoriametrics.com/#how-to-export-time-series) +- [`/api/v1/query`](https://docs.victoriametrics.com/url-examples/#apiv1query) when the `query` contains [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) ending with some duration in square brackets. For example, `/api/v1/query?query=up[1h]&time=2024-01-02T03:00:00Z`. - This query returns [raw samples](./keyConcepts.md#raw-samples) for [time series](./keyConcepts.md#time-series) + This query returns [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) with the `up` name on the time range `(2024-01-02T02:00:00 .. 2024-01-02T03:00:00]`. See [this article](https://valyala.medium.com/analyzing-prometheus-data-with-external-tools-5f3e5e147639) for details. @@ -631,9 +631,9 @@ Data replication can be used for increasing storage durability. See [these docs] ## Capacity planning -VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](./CaseStudies/README.md). +VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](https://docs.victoriametrics.com/casestudies/). -Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), [series churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](./README.md#troubleshooting). +Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series), [series churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](https://docs.victoriametrics.com/#troubleshooting). The needed storage space for the given retention (the retention is set via `-retentionPeriod` command-line flag at `vmstorage`) can be extrapolated from disk space usage in a test run. For example, if the storage space usage is 10GB after a day-long test run on a production workload, then it will need at least `10GB*100=1TB` of disk space for `-retentionPeriod=100d` (100-days retention period). Storage space usage can be monitored with [the official Grafana dashboard for VictoriaMetrics cluster](#monitoring). @@ -649,7 +649,7 @@ Some capacity planning tips for VictoriaMetrics cluster: - The [replication](#replication-and-data-safety) increases the amounts of needed resources for the cluster by up to `N` times where `N` is replication factor. This is because `vminsert` stores `N` copies of every ingested sample on distinct `vmstorage` nodes. These copies are de-duplicated by `vmselect` during querying. The most cost-efficient and performant solution for data durability is to rely on replicated durable persistent disks such as [Google Compute persistent disks](https://cloud.google.com/compute/docs/disks#pdspecs) instead of using the [replication at VictoriaMetrics level](#replication-and-data-safety). - It is recommended to run a cluster with big number of small `vmstorage` nodes instead of a cluster with small number of big `vmstorage` nodes. This increases chances that the cluster remains available and stable when some of `vmstorage` nodes are temporarily unavailable during maintenance events such as upgrades, configuration changes or migrations. For example, when a cluster contains 10 `vmstorage` nodes and a single node becomes temporarily unavailable, then the workload on the remaining 9 nodes increases by `1/9=11%`. When a cluster contains 3 `vmstorage` nodes and a single node becomes temporarily unavailable, then the workload on the remaining 2 nodes increases by `1/2=50%`. The remaining `vmstorage` nodes may have no enough free capacity for handling the increased workload. In this case the cluster may become overloaded, which may result to decreased availability and stability. -- Cluster capacity for [active time series](./FAQ.md#what-is-an-active-time-series) can be increased by increasing RAM and CPU resources per each `vmstorage` node or by adding new `vmstorage` nodes. +- Cluster capacity for [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) can be increased by increasing RAM and CPU resources per each `vmstorage` node or by adding new `vmstorage` nodes. - Query latency can be reduced by increasing CPU resources per each `vmselect` node, since each incoming query is processed by a single `vmselect` node. Performance for heavy queries scales with the number of available CPU cores at `vmselect` node, since `vmselect` processes time series referred by the query on all the available CPU cores. - If the cluster needs to process incoming queries at a high rate, then its capacity can be increased by adding more `vmselect` nodes, so incoming queries could be spread among bigger number of `vmselect` nodes. - By default `vminsert` compresses the data it sends to `vmstorage` in order to reduce network bandwidth usage. The compression takes additional CPU resources at `vminsert`. If `vminsert` nodes have limited CPU, then the compression can be disabled by passing `-rpc.disableCompression` command-line flag at `vminsert` nodes. @@ -685,64 +685,64 @@ Some workloads may need fine-grained resource usage limits. In these cases the f See also `-search.maxMemoryPerQuery` command-line flag at `vmselect`. - `-search.maxQueueDuration` at `vmselect` and `vmstorage` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests` concurrent queries are executed. -- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./README.md#prometheus-querying-api-enhancements) - query args at [/api/v1/labels](./url-examples.md#apiv1labels) and - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). +- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) + query args at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This may be useful for reducing the load on `vmstorage` if the provided extra filters match too many time series. The downside is that the endpoints can return labels and series, which do not match the provided extra filters. - `-search.maxSamplesPerSeries` at `vmselect` limits the number of raw samples the query can process per each time series. `vmselect` processes raw samples sequentially per each found time series during the query. It unpacks raw samples on the selected time range - per each time series into memory and then applies the given [rollup function](./MetricsQL.md#rollup-functions). + per each time series into memory and then applies the given [rollup function](https://docs.victoriametrics.com/metricsql/#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage at `vmselect` in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series. - `-search.maxSamplesPerQuery` at `vmselect` limits the number of raw samples a single query can process. This allows limiting CPU usage at `vmselect` for heavy queries. - `-search.maxResponseSeries` at `vmselect` limits the number of time series a single query can return - from [`/api/v1/query`](./keyConcepts.md#instant-query) and [`/api/v1/query_range`](./keyConcepts.md#range-query). + from [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each - matching time series from [range query](./keyConcepts.md#range-query). + matching time series from [range query](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsSubqueryPerTimeseries` limits the number of calculated points, which can be generated - per each matching time series during [subquery](./MetricsQL.md#subqueries) evaluation. -- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](./MetricsQL.md#aggregate-functions) in a single query. + per each matching time series during [subquery](https://docs.victoriametrics.com/metricsql/#subqueries) evaluation. +- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions) in a single query. - `-search.maxSeries` at `vmselect` limits the number of time series, which may be returned from - [/api/v1/series](./url-examples.md#apiv1series). + [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amounts of CPU time and memory at `vmstorage` and `vmselect` when the database contains - big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxSeries` to quite low value in order limit CPU and memory usage. - `-search.maxTagKeys` at `vmstorage` limits the number of items, which may be returned from - [/api/v1/labels](./url-examples.md#apiv1labels). This endpoint is used mostly by Grafana + [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory at `vmstorage` and `vmselect` - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. - `-search.maxTagValues` at `vmstorage` limits the number of items, which may be returned from - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). This endpoint is used mostly by Grafana + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory at `vmstorage` and `vmselect` - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. - `-search.maxLabelsAPISeries` at `vmselect` limits the number of time series, which can be scanned - when performing [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series) requests. + when performing [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) requests. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPISeries` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.ignoreExtraFiltersAtLabelsAPI`. -- `-search.maxLabelsAPIDuration` at `vmselect` limits the duration for requests to [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series). +- `-search.maxLabelsAPIDuration` at `vmselect` limits the duration for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). The limit can be altered for each query by passing `timeout` GET parameter, but can't exceed the limit specified via cmd-line flag. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPIDuration` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPISeries` and `-search.ignoreExtraFiltersAtLabelsAPI`. - `-storage.maxDailySeries` at `vmstorage` can be used for limiting the number of time series seen per day aka - [time series churn rate](./FAQ.md#what-is-high-churn-rate). See [cardinality limiter docs](#cardinality-limiter). -- `-storage.maxHourlySeries` at `vmstorage` can be used for limiting the number of [active time series](./FAQ.md#what-is-an-active-time-series). + [time series churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). See [cardinality limiter docs](#cardinality-limiter). +- `-storage.maxHourlySeries` at `vmstorage` can be used for limiting the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). See [cardinality limiter docs](#cardinality-limiter). -See also [capacity planning docs](#capacity-planning) and [cardinality limiter in vmagent](./vmagent.md#cardinality-limiter). +See also [capacity planning docs](#capacity-planning) and [cardinality limiter in vmagent](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). ## High availability @@ -755,8 +755,8 @@ across multiple availability zones, since cross-AZ network usually has lower ban error rates comparing the network inside a single AZ. If you need multi-AZ setup, then it is recommended running independent clusters in each AZ and setting up -[vmagent](./vmagent.md) in front of these clusters, so it could replicate incoming data -into all the cluster - see [these docs](./vmagent.md#multitenancy) for details. +[vmagent](https://docs.victoriametrics.com/vmagent/) in front of these clusters, so it could replicate incoming data +into all the cluster - see [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy) for details. Then an additional `vmselect` nodes can be configured for reading the data from multiple clusters according to [these docs](#multi-level-cluster-setup). @@ -779,7 +779,7 @@ The multi-level cluster setup for `vminsert` nodes has the following shortcoming * Data ingestion speed is limited by the slowest link to AZ. * `vminsert` nodes at top level re-route incoming data to the remaining AZs when some AZs are temporarily unavailable. This results in data gaps at AZs which were temporarily unavailable. -These issues are addressed by [vmagent](./vmagent.md) when it runs in [multitenancy mode](./vmagent.md#multitenancy). +These issues are addressed by [vmagent](https://docs.victoriametrics.com/vmagent/) when it runs in [multitenancy mode](https://docs.victoriametrics.com/vmagent/#multitenancy). `vmagent` buffers data, which must be sent to a particular AZ, when this AZ is temporarily unavailable. The buffer is stored on disk. The buffered data is sent to AZ as soon as it becomes available. ## vmstorage groups at vmselect @@ -865,7 +865,7 @@ for newly ingested data when `N-1` of storage nodes are unavailable. VictoriaMetrics stores timestamps with millisecond precision, so `-dedup.minScrapeInterval=1ms` command-line flag must be passed to `vmselect` nodes when the replication is enabled, so they could de-duplicate replicated samples obtained from distinct `vmstorage` nodes during querying. If duplicate data is pushed to VictoriaMetrics -from identically configured [vmagent](./vmagent.md) instances or Prometheus instances, then the `-dedup.minScrapeInterval` must be set +from identically configured [vmagent](https://docs.victoriametrics.com/vmagent/) instances or Prometheus instances, then the `-dedup.minScrapeInterval` must be set to `scrape_interval` from scrape configs according to [deduplication docs](#deduplication). Note that [replication doesn't save from disaster](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883), @@ -880,7 +880,7 @@ HDD-based persistent disks should be enough for the majority of use cases. It is ## Deduplication -Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](./README.md#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects: +Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](https://docs.victoriametrics.com/#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects: By default, `vminsert` tries to route all the samples for a single time series to a single `vmstorage` node. But samples for a single time series can be spread among multiple `vmstorage` nodes under certain conditions: * when adding/removing `vmstorage` nodes. Then new samples for a part of time series will be routed to another `vmstorage` nodes; @@ -896,7 +896,7 @@ for protecting from user errors such as accidental data deletion. The following steps must be performed for each `vmstorage` node for creating a backup: 1. Create an instant snapshot by navigating to `/snapshot/create` HTTP handler. It will create snapshot and return its name. -1. Archive the created snapshot from `<-storageDataPath>/snapshots/` folder using [vmbackup](./vmbackup.md). +1. Archive the created snapshot from `<-storageDataPath>/snapshots/` folder using [vmbackup](https://docs.victoriametrics.com/vmbackup/). The archival process doesn't interfere with `vmstorage` work, so it may be performed at any suitable time. 1. Delete unused snapshots via `/snapshot/delete?snapshot=` or `/snapshot/delete_all` in order to free up occupied storage space. @@ -905,13 +905,13 @@ There is no need in synchronizing backups among all the `vmstorage` nodes. Restoring from backup: 1. Stop `vmstorage` node with `kill -INT`. -1. Restore data from backup using [vmrestore](./vmrestore.md) into `-storageDataPath` directory. +1. Restore data from backup using [vmrestore](https://docs.victoriametrics.com/vmrestore/) into `-storageDataPath` directory. 1. Start `vmstorage` node. ## Retention filters -[VictoriaMetrics enterprise](./enterprise.md) supports configuring multiple retentions for distinct sets of time series -by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](./README.md#retention-filters) for details on this feature. +[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) supports configuring multiple retentions for distinct sets of time series +by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](https://docs.victoriametrics.com/#retention-filters) for details on this feature. Additionally, enterprise version of VictoriaMetrics cluster supports multiple retentions for distinct sets of [tenants](#multitenancy) by specifying filters on `vm_account_id` and/or `vm_project_id` pseudo-labels in `-retentionFilter` command-line flag. @@ -932,7 +932,7 @@ For example, the following config sets retention to 5 days for time series with -retentionFilter='{vm_account_id="5",env="dev"}:5d' ``` -See also [these docs](./README.md#retention-filters) for additional details on retention filters. +See also [these docs](https://docs.victoriametrics.com/#retention-filters) for additional details on retention filters. See also [downsampling](#downsampling). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). @@ -940,10 +940,10 @@ See how to request a free trial license [here](https://victoriametrics.com/produ ## Downsampling -Downsampling is available in [enterprise version of VictoriaMetrics](./enterprise.md). -It is configured with `-downsampling.period` command-line flag according to [these docs](./README.md#downsampling). +Downsampling is available in [enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/). +It is configured with `-downsampling.period` command-line flag according to [these docs](https://docs.victoriametrics.com/#downsampling). -It is possible to downsample series, which belong to a particular [tenant](#multitenancy) by using [filters](./keyConcepts.md#filtering) +It is possible to downsample series, which belong to a particular [tenant](#multitenancy) by using [filters](https://docs.victoriametrics.com/keyconcepts/#filtering) on `vm_account_id` or `vm_project_id` pseudo-labels in `-downsampling.period` command-line flag. For example, the following config leaves the last sample per each minute for samples older than one hour only for [tenants](#multitenancy) with accountID equal to 12 and 42, while series for other tenants are dropped: @@ -962,7 +962,7 @@ The same flag value must be passed to both `vmstorage` and `vmselect` nodes. Con command-line flag makes query results more consistent, because `vmselect` uses the maximum configured downsampling interval on the requested time range if this time range covers multiple downsampling levels. For example, if `-downsampling.period=30d:5m` and the query requests the last 60 days of data, then `vmselect` -downsamples all the [raw samples](./keyConcepts.md#raw-samples) on the requested time range +downsamples all the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the requested time range using 5 minute interval. If `-downsampling.period` command-line flag isn't set at `vmselect`, then query results can be less consistent because of mixing raw and downsampled data. @@ -999,7 +999,7 @@ It is safe sharing the collected profiles from security point of view, since the ## vmalert -vmselect is capable of proxying requests to [vmalert](./vmalert.md) +vmselect is capable of proxying requests to [vmalert](https://docs.victoriametrics.com/vmalert/) when `-vmalert.proxyURL` flag is set. Use this feature for the following cases: * for proxying requests from [Grafana Alerting UI](https://grafana.com/docs/grafana/latest/alerting/); * for accessing vmalerts UI through vmselects Web interface. @@ -1021,7 +1021,7 @@ Feel free asking any questions regarding VictoriaMetrics: * [Google groups](https://groups.google.com/forum/#!forum/victorametrics-users) * [Mastodon](https://mastodon.social/@victoriametrics/) -If you like VictoriaMetrics and want contributing, then please read [these docs](./CONTRIBUTING.md). +If you like VictoriaMetrics and want contributing, then please read [these docs](https://docs.victoriametrics.com/contributing/). ## Reporting bugs @@ -1043,19 +1043,19 @@ Below is the output for `/path/to/vminsert -help`: -cacheExpireDuration duration Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s) -cluster.tls - Whether to use TLS for connections to -storageNode. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCAFile string - Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCertFile string - Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsInsecureSkipVerify - Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsKeyFile string - Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.vminsertConnsShutdownDuration duration - The time needed for gradual closing of upstream vminsert connections during graceful shutdown. Bigger duration reduces spikes in CPU, RAM and disk IO load on the remaining lower-level clusters during rolling restart. Smaller duration reduces the time needed to close all the upstream vminsert connections, thus reducing the time for graceful shutdown. See {{% ref "./Cluster-VictoriaMetrics.md#improving-re-routing-performance-during-restart" %}} (default 25s) + The time needed for gradual closing of upstream vminsert connections during graceful shutdown. Bigger duration reduces spikes in CPU, RAM and disk IO load on the remaining lower-level clusters during rolling restart. Smaller duration reduces the time needed to close all the upstream vminsert connections, thus reducing the time for graceful shutdown. See https://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart (default 25s) -clusternativeListenAddr string - TCP address to listen for data from other vminsert nodes in multi-level cluster setup. See {{% ref "./Cluster-VictoriaMetrics.md#multi-level-cluster-setup" %}}. Usually :8400 should be set to match default vmstorage port for vminsert. Disabled work if empty + TCP address to listen for data from other vminsert nodes in multi-level cluster setup. See https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup . Usually :8400 should be set to match default vmstorage port for vminsert. Disabled work if empty -csvTrimTimestamp duration Trim timestamps when importing csv data to this duration. Minimum practical duration is 1ms. Higher duration (i.e. 1s) may be used for reducing disk space usage for timestamp data (default 1ms) -datadog.maxInsertRequestSize size @@ -1064,7 +1064,7 @@ Below is the output for `/path/to/vminsert -help`: -datadog.sanitizeMetricName Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true) -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} + Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing -disableRerouting Whether to disable re-routing when some of vmstorage nodes accept incoming data at slower speed compared to other storage nodes. Disabled re-routing limits the ingestion rate by the slowest vmstorage node. On the other side, disabled re-routing minimizes the number of active time series in the cluster during rolling restarts and during spikes in series churn rate. See also -disableReroutingOnUnavailable and -dropSamplesOnOverload (default true) -disableReroutingOnUnavailable @@ -1074,11 +1074,11 @@ Below is the output for `/path/to/vminsert -help`: -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula - Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -flagsAuthKey value @@ -1087,7 +1087,7 @@ Below is the output for `/path/to/vminsert -help`: -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -graphite.sanitizeMetricName - Sanitize metric names for the ingested Graphite data. See {{% ref "./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}} + Sanitize metric names for the ingested Graphite data. See https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd -graphiteListenAddr string TCP and UDP address to listen for Graphite plaintext data. Usually :2003 must be set. Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol -graphiteListenAddr.useProxyProtocol @@ -1201,18 +1201,18 @@ Below is the output for `/path/to/vminsert -help`: Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array - Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array - Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -newrelic.maxInsertRequestSize size The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864) -opentelemetry.usePrometheusNaming - Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./README.md#sending-data-via-opentelemetry" %}} + Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see https://docs.victoriametrics.com/#sending-data-via-opentelemetry -opentsdbHTTPListenAddr string TCP address to listen for OpenTSDB HTTP put requests. Usually :4242 must be set. Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol -opentsdbHTTPListenAddr.useProxyProtocol @@ -1246,11 +1246,11 @@ Below is the output for `/path/to/vminsert -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -relabelConfig string - Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./README.md#relabeling" %}} for details. The config is reloaded on SIGHUP signal + Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal -relabelConfigCheckInterval duration Interval for checking for changes in '-relabelConfig' file. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes -replicationFactor int @@ -1260,23 +1260,23 @@ Below is the output for `/path/to/vminsert -help`: -sortLabels Whether to sort labels for incoming samples before writing them to storage. This may be needed for reducing memory usage at storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}. Enabled sorting for labels can slow down ingestion performance a bit -storageNode array - Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via DNS SRV records. For example, -storageNode=srv+vmstorage.addrs . See {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}} + Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via DNS SRV records. For example, -storageNode=srv+vmstorage.addrs . See https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -storageNode.discoveryInterval duration - Interval for refreshing -storageNode list behind DNS SRV records. The minimum supported interval is 1s. See {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} (default 2s) + Interval for refreshing -storageNode list behind DNS SRV records. The minimum supported interval is 1s. See https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ (default 2s) -storageNode.filter string - An optional regexp filter for discovered -storageNode addresses according to {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}}. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + An optional regexp filter for discovered -storageNode addresses according to https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -tls array Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string - Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string - Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array - Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array @@ -1317,15 +1317,15 @@ Below is the output for `/path/to/vmselect -help`: -cacheExpireDuration duration Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s) -cluster.tls - Whether to use TLS for connections to -storageNode. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCAFile string - Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCertFile string - Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsInsecureSkipVerify - Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsKeyFile string - Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.disableCompression Whether to disable compression of the data sent to vmselect via -clusternativeListenAddr. This reduces CPU usage at the cost of higher network bandwidth usage -clusternative.maxConcurrentRequests int @@ -1339,40 +1339,40 @@ Below is the output for `/path/to/vmselect -help`: -clusternative.maxTagValues int The maximum number of tag values returned per search at -clusternativeListenAddr (default 100000) -clusternative.tls - Whether to use TLS when accepting connections at -clusternativeListenAddr. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to use TLS when accepting connections at -clusternativeListenAddr. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.tlsCAFile string - Path to TLS CA file to use for verifying certificates provided by vmselect, which connects at -clusternativeListenAddr if -clusternative.tls flag is set. By default system CA is used. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to TLS CA file to use for verifying certificates provided by vmselect, which connects at -clusternativeListenAddr if -clusternative.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.tlsCertFile string - Path to server-side TLS certificate file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to server-side TLS certificate file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.tlsCipherSuites array - Optional list of TLS cipher suites used for connections at -clusternativeListenAddr if -clusternative.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Optional list of TLS cipher suites used for connections at -clusternativeListenAddr if -clusternative.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -clusternative.tlsInsecureSkipVerify - Whether to skip verification of TLS certificates provided by vmselect, which connects to -clusternativeListenAddr if -clusternative.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to skip verification of TLS certificates provided by vmselect, which connects to -clusternativeListenAddr if -clusternative.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternative.tlsKeyFile string - Path to server-side TLS key file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to server-side TLS key file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -clusternativeListenAddr string - TCP address to listen for requests from other vmselect nodes in multi-level cluster setup. See {{% ref "./Cluster-VictoriaMetrics.md#multi-level-cluster-setup" %}}. Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty + TCP address to listen for requests from other vmselect nodes in multi-level cluster setup. See https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup . Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication for details -deleteAuthKey value authKey for metrics' deletion via /prometheus/api/v1/admin/tsdb/delete_series and /graphite/tags/delSeries Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} + Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula - Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -flagsAuthKey value @@ -1381,7 +1381,7 @@ Below is the output for `/path/to/vmselect -help`: -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -globalReplicationFactor int - How many copies of every ingested sample is available across vmstorage groups. vmselect continues returning full responses when up to globalReplicationFactor-1 vmstorage groups are temporarily unavailable. See {{% ref "./Cluster-VictoriaMetrics.md#vmstorage-groups-at-vmselect" %}}. See also -replicationFactor (default 1) + How many copies of every ingested sample is available across vmstorage groups. vmselect continues returning full responses when up to globalReplicationFactor-1 vmstorage groups are temporarily unavailable. See https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect . See also -replicationFactor (default 1) -http.connTimeout duration Incoming connections to -httpListenAddr are closed after the configured timeout. This may help evenly spreading load among a cluster of services behind TCP-level load balancer. Zero value disables closing of incoming connections (default 2m0s) -http.disableResponseCompression @@ -1454,11 +1454,11 @@ Below is the output for `/path/to/vmselect -help`: Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array - Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array - Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -pprofAuthKey value @@ -1479,7 +1479,7 @@ Below is the output for `/path/to/vmselect -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -replicationFactor array @@ -1490,9 +1490,9 @@ Below is the output for `/path/to/vmselect -help`: -search.denyPartialResponse Whether to deny partial responses if a part of -storageNode instances fail to perform queries; this trades availability over consistency; see also -search.maxQueryDuration -search.disableCache - Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./README.md#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup + Whether to disable response caching. This may be useful when ingesting historical data. See https://docs.victoriametrics.com/#backfilling . See also -search.resetRollupResultCacheOnStartup -search.disableImplicitConversion - Whether to return an error for queries that rely on implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. See also -search.logImplicitConversion + Whether to return an error for queries that rely on implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. See also -search.logImplicitConversion -search.graphiteMaxPointsPerSeries int The maximum number of points per series Graphite render API can return (default 1000000) -search.graphiteStorageStep duration @@ -1502,7 +1502,7 @@ Below is the output for `/path/to/vmselect -help`: -search.latencyOffset duration The time when data points become visible in query results after the collection. It can be overridden on per-query basis via latency_offset arg. Too small value can result in incomplete last points for query results (default 30s) -search.logImplicitConversion - Whether to log queries with implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. Such conversion can be disabled using -search.disableImplicitConversion + Whether to log queries with implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. Such conversion can be disabled using -search.disableImplicitConversion -search.logQueryMemoryUsage size Log query and increment vm_memory_intensive_queries_total metric each time the query requires more memory than specified by this flag. This may help detecting and optimizing heavy queries. Query logging is disabled by default. See also -search.logSlowQueryDuration and -search.maxMemoryPerQuery Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) @@ -1517,11 +1517,11 @@ Below is the output for `/path/to/vmselect -help`: -search.maxFederateSeries int The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000) -search.maxGraphiteSeries int - The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./README.md#graphite-render-api-usage" %}} (default 300000) + The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000) -search.maxGraphiteTagKeys int - The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag keys returned from Graphite API, which returns tags. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxGraphiteTagValues int - The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag values returned from Graphite API, which returns tag values. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxLabelsAPIDuration duration The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s) -search.maxLabelsAPISeries int @@ -1581,7 +1581,7 @@ Below is the output for `/path/to/vmselect -help`: Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path -search.resetRollupResultCacheOnStartup - Whether to reset rollup result cache on startup. See {{% ref "./README.md#rollup-result-cache" %}}. See also -search.disableCache + Whether to reset rollup result cache on startup. See https://docs.victoriametrics.com/#rollup-result-cache . See also -search.disableCache -search.setLookbackToStep Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored -search.skipSlowReplicas @@ -1593,23 +1593,23 @@ Below is the output for `/path/to/vmselect -help`: Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -storageNode array - Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via DNS SRV records. For example, -storageNode=srv+vmstorage.addrs . See {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}} + Comma-separated addresses of vmstorage nodes; usage: -storageNode=vmstorage-host1,...,vmstorage-hostN . Enterprise version of VictoriaMetrics supports automatic discovery of vmstorage addresses via DNS SRV records. For example, -storageNode=srv+vmstorage.addrs . See https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -storageNode.discoveryInterval duration - Interval for refreshing -storageNode list behind DNS SRV records. The minimum supported interval is 1s. See {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} (default 2s) + Interval for refreshing -storageNode list behind DNS SRV records. The minimum supported interval is 1s. See https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ (default 2s) -storageNode.filter string - An optional regexp filter for discovered -storageNode addresses according to {{% ref "./Cluster-VictoriaMetrics.md#automatic-vmstorage-discovery" %}}. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + An optional regexp filter for discovered -storageNode addresses according to https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -tls array Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string - Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string - Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array - Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array @@ -1654,37 +1654,37 @@ Below is the output for `/path/to/vmstorage -help`: -cacheExpireDuration duration Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s) -cluster.tls - Whether to use TLS when accepting connections from vminsert and vmselect. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to use TLS when accepting connections from vminsert and vmselect. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCAFile string - Path to TLS CA file to use for verifying certificates provided by vminsert and vmselect if -cluster.tls flag is set. By default system CA is used. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to TLS CA file to use for verifying certificates provided by vminsert and vmselect if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCertFile string - Path to server-side TLS certificate file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to server-side TLS certificate file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsCipherSuites array - Optional list of TLS cipher suites used for connections from vminsert and vmselect if -cluster.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Optional list of TLS cipher suites used for connections from vminsert and vmselect if -cluster.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -cluster.tlsInsecureSkipVerify - Whether to skip verification of TLS certificates provided by vminsert and vmselect if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Whether to skip verification of TLS certificates provided by vminsert and vmselect if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -cluster.tlsKeyFile string - Path to server-side TLS key file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Path to server-side TLS key file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication for details -denyQueriesOutsideRetention Whether to deny queries outside of the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} + Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula - Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -finalMergeDelay duration @@ -1780,11 +1780,11 @@ Below is the output for `/path/to/vmstorage -help`: Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array - Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array - Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -pprofAuthKey value @@ -1807,11 +1807,11 @@ Below is the output for `/path/to/vmstorage -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionFilter array - Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./README.md#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionPeriod value @@ -1844,26 +1844,26 @@ Below is the output for `/path/to/vmstorage -help`: Automatically delete snapshots older than -snapshotsMaxAge if it is set to non-zero duration. Make sure that backup process has enough time to finish the backup before the corresponding snapshot is automatically deleted The following optional suffixes are supported: s (second), m (minute), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 0) -storage.cacheSizeIndexDBDataBlocks size - Overrides max size for indexdb/dataBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBIndexBlocks size - Overrides max size for indexdb/indexBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBTagFilters size - Overrides max size for indexdb/tagFiltersToMetricIDs cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeStorageTSID size - Overrides max size for storage/tsid cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.maxDailySeries int - The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxHourlySeries + The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxHourlySeries -storage.maxHourlySeries int - The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxDailySeries + The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxDailySeries -storage.minFreeDiskSpaceBytes size The minimum free disk space at -storageDataPath after which the storage stops accepting new data Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 10000000) -storage.vminsertConnsShutdownDuration duration - The time needed for gradual closing of vminsert connections during graceful shutdown. Bigger duration reduces spikes in CPU, RAM and disk IO load on the remaining vmstorage nodes during rolling restart. Smaller duration reduces the time needed to close all the vminsert connections, thus reducing the time for graceful shutdown. See {{% ref "./Cluster-VictoriaMetrics.md#improving-re-routing-performance-during-restart" %}} (default 25s) + The time needed for gradual closing of vminsert connections during graceful shutdown. Bigger duration reduces spikes in CPU, RAM and disk IO load on the remaining vmstorage nodes during rolling restart. Smaller duration reduces the time needed to close all the vminsert connections, thus reducing the time for graceful shutdown. See https://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart (default 25s) -storageDataPath string Path to storage data (default "vmstorage-data") -tls array @@ -1871,11 +1871,11 @@ Below is the output for `/path/to/vmstorage -help`: Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string - Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string - Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array - Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array diff --git a/docs/VictoriaLogs/CHANGELOG.md b/docs/VictoriaLogs/CHANGELOG.md index 7e0aabc86..4705bc419 100644 --- a/docs/VictoriaLogs/CHANGELOG.md +++ b/docs/VictoriaLogs/CHANGELOG.md @@ -12,7 +12,7 @@ aliases: - /VictoriaLogs/CHANGELOG.html --- The following `tip` changes can be tested by building VictoriaLogs from the latest commit of [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/) repository -according to [these docs](./VictoriaLogs/QuickStart.md#building-from-source-code) +according to [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/#building-from-source-code) ## tip @@ -20,21 +20,21 @@ according to [these docs](./VictoriaLogs/QuickStart.md#building-from-source-code Released at 2024-07-10 -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): show a spinner on top of bar chart until user's request is finished. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558). -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): use compact representation of JSON lines at `JSON` tab if only a single [log field](./VictoriaLogs/keyConcepts.md#data-model) is queried. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6559). -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): properly show the number of matching logs on the selected time range at bar chart for queries with arbitrary [pipes](./VictoriaLogs/LogsQL.md#pipes), including [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe) and [`top` pipe](./VictoriaLogs/LogsQL.md#top-pipe). +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): show a spinner on top of bar chart until user's request is finished. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558). +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): use compact representation of JSON lines at `JSON` tab if only a single [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) is queried. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6559). +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): properly show the number of matching logs on the selected time range at bar chart for queries with arbitrary [pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes), including [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) and [`top` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#top-pipe). ## [v0.27.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.27.1-victorialogs) Released at 2024-07-05 -* BUGFIX: properly JSON-encode strings with special chars in [HTTP querying API](./VictoriaLogs/querying/README.md#http-api) responses. This fixes the `error decode response: invalid character 'x' in string escape code` error in [VictoriaLogs datasource for Grafana](https://github.com/VictoriaMetrics/victorialogs-datasource/). See [this issue](https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24). The issue has been introduced in the release [v0.9.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.0-victorialogs). +* BUGFIX: properly JSON-encode strings with special chars in [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api) responses. This fixes the `error decode response: invalid character 'x' in string escape code` error in [VictoriaLogs datasource for Grafana](https://github.com/VictoriaMetrics/victorialogs-datasource/). See [this issue](https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24). The issue has been introduced in the release [v0.9.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.0-victorialogs). ## [v0.27.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.27.0-victorialogs) Released at 2024-07-02 -* FEATURE: add `-syslog.useLocalTimestamp.tcp` and `-syslog.useLocalTimestamp.udp` command-line flags, which could be used for using the local timestamp as [`_time` field](./VictoriaLogs/keyConcepts.md#time-field) for the logs ingested via the corresponding `-syslog.listenAddr.tcp` / `-syslog.listenAddr.udp`. By default the timestap from the syslog message is used as [`_time` field](./VictoriaLogs/keyConcepts.md#time-field). See [these docs](./VictoriaLogs/data-ingestion/syslog.md). +* FEATURE: add `-syslog.useLocalTimestamp.tcp` and `-syslog.useLocalTimestamp.udp` command-line flags, which could be used for using the local timestamp as [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) for the logs ingested via the corresponding `-syslog.listenAddr.tcp` / `-syslog.listenAddr.udp`. By default the timestap from the syslog message is used as [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field). See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/). * BUGFIX: make slowly ingested logs visible for search as soon as they are ingested into VictoriaLogs. Previously slowly ingested logs could remain invisible for search for long time. @@ -42,89 +42,89 @@ Released at 2024-07-02 Released at 2024-07-01 -* BUGFIX: return the proper surrounding logs for [`stream_context` pipe](./VictoriaLogs/LogsQL.md#stream_context-pipe) when additional [pipes](./VictoriaLogs/LogsQL.md#pipes) are put after the `stream_context` pipe. This has been broken in [v0.26.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.26.0-victorialogs). +* BUGFIX: return the proper surrounding logs for [`stream_context` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stream_context-pipe) when additional [pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes) are put after the `stream_context` pipe. This has been broken in [v0.26.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.26.0-victorialogs). ## [v0.26.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.26.0-victorialogs) Released at 2024-07-01 -* FEATURE: add ability to return log position (aka rank) after sorting logs with [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe). This can be done by adding `rank as ` to the end of `| sort ...` pipe. For example, `_time:5m | sort by (_time) rank as position` instructs storing position of every sorted log line into `position` [field name](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: add delimiter log with `---` message between log chunks returned by [`stream_context` pipe](./VictoriaLogs/LogsQL.md#stream_context-pipe). This should simplify investigation of the returned logs. -* FEATURE: reduce memory usage when big number of context logs are requested from [`stream_context` pipe](./VictoriaLogs/LogsQL.md#stream_context-pipe). +* FEATURE: add ability to return log position (aka rank) after sorting logs with [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). This can be done by adding `rank as ` to the end of `| sort ...` pipe. For example, `_time:5m | sort by (_time) rank as position` instructs storing position of every sorted log line into `position` [field name](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: add delimiter log with `---` message between log chunks returned by [`stream_context` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stream_context-pipe). This should simplify investigation of the returned logs. +* FEATURE: reduce memory usage when big number of context logs are requested from [`stream_context` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stream_context-pipe). ## [v0.25.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.25.0-victorialogs) Released at 2024-06-28 -* FEATURE: add ability to select surrounding logs in front and after the selected logs via [`stream_context` pipe](./VictoriaLogs/LogsQL.md#stream_context-pipe). This functionality may be useful for investigating stacktraces, panics or some correlated log messages. This functionality is similar to `grep -A` and `grep -B`. -* FEATURE: add ability to return top `N` `"fields"` groups from [`/select/logsql/hits` HTTP endpoint](./VictoriaLogs/querying/README.md#querying-hits-stats), by specifying `fields_limit=N` query arg. This query arg is going to be used in [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545). +* FEATURE: add ability to select surrounding logs in front and after the selected logs via [`stream_context` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stream_context-pipe). This functionality may be useful for investigating stacktraces, panics or some correlated log messages. This functionality is similar to `grep -A` and `grep -B`. +* FEATURE: add ability to return top `N` `"fields"` groups from [`/select/logsql/hits` HTTP endpoint](https://docs.victoriametrics.com/victorialogs/querying/#querying-hits-stats), by specifying `fields_limit=N` query arg. This query arg is going to be used in [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545). -* BUGFIX: fix `runtime error: index out of range [0] with length 0` panic when empty lines are ingested via [Syslog format](./VictoriaLogs/data-ingestion/syslog.md) by Cisco controllers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548). +* BUGFIX: fix `runtime error: index out of range [0] with length 0` panic when empty lines are ingested via [Syslog format](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/) by Cisco controllers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548). ## [v0.24.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.24.0-victorialogs) Released at 2024-06-27 -* FEATURE: add `/select/logsql/tail` HTTP endpoint, which can be used for live tailing of [LogsQL query](./VictoriaLogs/LogsQL.md) results. See [these docs](./VictoriaLogs/querying/README.md#live-tailing) for details. -* FEATURE: add `/select/logsql/stream_ids` HTTP endpoint, which can be used for returning [`_stream_id` values](./VictoriaLogs/keyConcepts.md#stream-fields) with the number of hits for the given [LogsQL query](./VictoriaLogs/LogsQL.md). See [these docs](./VictoriaLogs/querying/README.md#querying-stream_ids) for details. -* FEATURE: add `-retention.maxDiskSpaceUsageBytes` command-line flag, which allows limiting disk space usage for [VictoriaLogs data](./VictoriaLogs/README.md#storage) by automatic dropping the oldest per-day partitions if the storage disk space usage becomes bigger than the `-retention.maxDiskSpaceUsageBytes`. See [these docs](./VictoriaLogs/README.md#retention-by-disk-space-usage). +* FEATURE: add `/select/logsql/tail` HTTP endpoint, which can be used for live tailing of [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/) results. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#live-tailing) for details. +* FEATURE: add `/select/logsql/stream_ids` HTTP endpoint, which can be used for returning [`_stream_id` values](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with the number of hits for the given [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/). See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-stream_ids) for details. +* FEATURE: add `-retention.maxDiskSpaceUsageBytes` command-line flag, which allows limiting disk space usage for [VictoriaLogs data](https://docs.victoriametrics.com/victorialogs/#storage) by automatic dropping the oldest per-day partitions if the storage disk space usage becomes bigger than the `-retention.maxDiskSpaceUsageBytes`. See [these docs](https://docs.victoriametrics.com/victorialogs/#retention-by-disk-space-usage). * BUGFIX: properly take into account query timeout specified via `-search.maxQueryDuration` command-line flag and/or via `timeout` query arg. Previously these timeouts could be ignored during query execution. -* BUGFIX: [web UI](./VictoriaLogs/querying/README.md#web-ui): fix the update of the relative time range when `Execute Query` is clicked. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6345). +* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): fix the update of the relative time range when `Execute Query` is clicked. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6345). ## [v0.23.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.23.0-victorialogs) Released at 2024-06-25 -* FEATURE: [syslog data ingestion](./VictoriaLogs/data-ingestion/syslog.md): parse [STRUCTURED-DATA](https://datatracker.ietf.org/doc/html/rfc5424#section-6.3) into `SD-ID.field1=value1`, `SD-ID.field2=value2`, ..., `SD-ID.fieldN=valueN` [log fields](./VictoriaLogs/keyConcepts.md#data-model). Previously the `STRUCTURED-DATA` was parsed into a single log field with the `SD-ID` name and `field1=value1 field2=value2 ... fieldN=valueN` value. This could complicate querying of such data. +* FEATURE: [syslog data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/): parse [STRUCTURED-DATA](https://datatracker.ietf.org/doc/html/rfc5424#section-6.3) into `SD-ID.field1=value1`, `SD-ID.field2=value2`, ..., `SD-ID.fieldN=valueN` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). Previously the `STRUCTURED-DATA` was parsed into a single log field with the `SD-ID` name and `field1=value1 field2=value2 ... fieldN=valueN` value. This could complicate querying of such data. -* BUGFIX: properly parse timestamps with timezones during [data ingestion](./VictoriaLogs/data-ingestion/README.md) and [querying](./VictoriaLogs/querying/README.md). This has been broken in [v0.20.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.0-victorialogs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508). +* BUGFIX: properly parse timestamps with timezones during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/victorialogs/querying/). This has been broken in [v0.20.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.0-victorialogs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508). ## [v0.22.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.22.0-victorialogs) Released at 2024-06-24 -* FEATURE: allow specifying multiple `_stream_id` values in [`_stream_id` filter](./VictoriaLogs/LogsQL.md#_stream_id-filter) via `_stream_id:in(id1, ..., idN)` syntax. -* FEATURE: allow specifying subquery for searching for `_stream_id` values inside [`_stream_id` filter](./VictoriaLogs/LogsQL.md#_stream_id-filter). For example, `_stream_id:in(_time:5m error | fields _stream_id)` returns logs for [logs streams](./VictoriaLogs/keyConcepts.md#stream-fields) with the `error` word across logs for the last 5 minutes. +* FEATURE: allow specifying multiple `_stream_id` values in [`_stream_id` filter](https://docs.victoriametrics.com/victorialogs/logsql/#_stream_id-filter) via `_stream_id:in(id1, ..., idN)` syntax. +* FEATURE: allow specifying subquery for searching for `_stream_id` values inside [`_stream_id` filter](https://docs.victoriametrics.com/victorialogs/logsql/#_stream_id-filter). For example, `_stream_id:in(_time:5m error | fields _stream_id)` returns logs for [logs streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with the `error` word across logs for the last 5 minutes. ## [v0.21.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.21.0-victorialogs) Released at 2024-06-20 -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): add a bar chart displaying the number of log entries over a time range. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6404). -* FEATURE: expose `_stream_id` field, which uniquely identifies [log streams](./VictoriaLogs/keyConcepts.md#stream-fields). This field can be used for quick obtaining of all the logs belonging to a particular stream via [`_stream_id` filter](./VictoriaLogs/LogsQL.md#_stream_id-filter). +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): add a bar chart displaying the number of log entries over a time range. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6404). +* FEATURE: expose `_stream_id` field, which uniquely identifies [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). This field can be used for quick obtaining of all the logs belonging to a particular stream via [`_stream_id` filter](https://docs.victoriametrics.com/victorialogs/logsql/#_stream_id-filter). ## [v0.20.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.2-victorialogs) Released at 2024-06-18 -* BUGFIX: properly parse timestamps with nanosecond precision for logs ingested via [jsonline format](./VictoriaLogs/data-ingestion/README.md#json-stream-api). The bug has been introduced in [v0.20.0 release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.0-victorialogs). +* BUGFIX: properly parse timestamps with nanosecond precision for logs ingested via [jsonline format](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api). The bug has been introduced in [v0.20.0 release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.0-victorialogs). ## [v0.20.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.1-victorialogs) Released at 2024-06-18 -* FEATURE: allow configuring multiple receivers with distinct configs for syslog messages. See [these docs](./VictoriaLogs/data-ingestion/syslog.md#multiple-configs). +* FEATURE: allow configuring multiple receivers with distinct configs for syslog messages. See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#multiple-configs). -* BUGFIX: properly read syslog messages over TCP and TLS connections according to [RFC5425](https://datatracker.ietf.org/doc/html/rfc5425) when [data ingestion for syslog protocol](./VictoriaLogs/data-ingestion/syslog.md) is enabled. +* BUGFIX: properly read syslog messages over TCP and TLS connections according to [RFC5425](https://datatracker.ietf.org/doc/html/rfc5425) when [data ingestion for syslog protocol](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/) is enabled. ## [v0.20.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.20.0-victorialogs) Released at 2024-06-17 -* FEATURE: add ability to accept logs in [Syslog format](https://en.wikipedia.org/wiki/Syslog). See [these docs](./VictoriaLogs/data-ingestion/syslog.md). -* FEATURE: add ability to specify timezone offset when parsing [rfc3164](https://datatracker.ietf.org/doc/html/rfc3164) syslog messages with [`unpack_syslog` pipe](./VictoriaLogs/LogsQL.md#unpack_syslog-pipe). -* FEATURE: add [`top` pipe](./VictoriaLogs/LogsQL.md#top-pipe) for returning top N sets of the given fields with the maximum number of matching log entries. +* FEATURE: add ability to accept logs in [Syslog format](https://en.wikipedia.org/wiki/Syslog). See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/). +* FEATURE: add ability to specify timezone offset when parsing [rfc3164](https://datatracker.ietf.org/doc/html/rfc3164) syslog messages with [`unpack_syslog` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_syslog-pipe). +* FEATURE: add [`top` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#top-pipe) for returning top N sets of the given fields with the maximum number of matching log entries. ## [v0.19.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.19.0-victorialogs) Released at 2024-06-11 -* FEATURE: do not allow starting the [filter](./VictoriaLogs/LogsQL.md#filters) with [pipe names](./VictoriaLogs/LogsQL.md#pipes) and [stats function names](./VictoriaLogs/LogsQL.md#stats-pipe-functions). This prevents from unexpected results returned by incorrect queries, which miss mandatory [filter](./VictoriaLogs/LogsQL.md#query-syntax). -* FEATURE: treat unexpected syslog message as [RFC3164](https://datatracker.ietf.org/doc/html/rfc3164) containing only the `message` field when using [`unpack_syslog` pipe](./VictoriaLogs/LogsQL.md#unpack_syslog-pipe). -* FEATURE: allow using `where` prefix instead of `filter` prefix in [`filter` pipe](./VictoriaLogs/LogsQL.md#filter-pipe). -* FEATURE: disallow unescaped `!` char in [LogsQL](./VictoriaLogs/LogsQL.md) queries, since it permits writing incorrect query, which may look like correct one. For example, `foo!:bar` instead of `foo:!bar`. -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): add markdown support to the `Group` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6292). +* FEATURE: do not allow starting the [filter](https://docs.victoriametrics.com/victorialogs/logsql/#filters) with [pipe names](https://docs.victoriametrics.com/victorialogs/logsql/#pipes) and [stats function names](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions). This prevents from unexpected results returned by incorrect queries, which miss mandatory [filter](https://docs.victoriametrics.com/victorialogs/logsql/#query-syntax). +* FEATURE: treat unexpected syslog message as [RFC3164](https://datatracker.ietf.org/doc/html/rfc3164) containing only the `message` field when using [`unpack_syslog` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_syslog-pipe). +* FEATURE: allow using `where` prefix instead of `filter` prefix in [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe). +* FEATURE: disallow unescaped `!` char in [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) queries, since it permits writing incorrect query, which may look like correct one. For example, `foo!:bar` instead of `foo:!bar`. +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): add markdown support to the `Group` view. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6292). * BUGFIX: return back the improved performance for queries with `*` filters (aka `SELECT *`). This has been broken in [v0.16.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.16.0-victorialogs). @@ -132,42 +132,42 @@ Released at 2024-06-11 Released at 2024-06-06 -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): improve displaying of logs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419) and the following issues: [6408](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408), [6405](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405), [6406](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406) and [6407](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407). -* FEATURE: add support for [day range filter](./VictoriaLogs/LogsQL.md#day-range-filter) and [week range filter](./VictoriaLogs/LogsQL.md#week-range-filter). These filters allow selecting logs on a particular time range per every day or on a particular day of the week. -* FEATURE: allow using `eval` instead of `math` keyword in [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): improve displaying of logs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419) and the following issues: [6408](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408), [6405](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405), [6406](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406) and [6407](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407). +* FEATURE: add support for [day range filter](https://docs.victoriametrics.com/victorialogs/logsql/#day-range-filter) and [week range filter](https://docs.victoriametrics.com/victorialogs/logsql/#week-range-filter). These filters allow selecting logs on a particular time range per every day or on a particular day of the week. +* FEATURE: allow using `eval` instead of `math` keyword in [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). ## [v0.17.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.17.0-victorialogs) Released at 2024-06-05 -* FEATURE: add [`pack_logfmt` pipe](./VictoriaLogs/LogsQL.md#pack_logfmt-pipe) for formatting [log fields](./VictoriaLogs/keyConcepts.md#data-model) into [logfmt](https://brandur.org/logfmt) messages. -* FEATURE: allow using IPv4 addresses in [range comparison filters](./VictoriaLogs/LogsQL.md#range-comparison-filter). For example, `ip:>'12.34.56.78'` is valid filter now. -* FEATURE: add `ceil()` and `floor()` functions to [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). -* FEATURE: add support for bitwise `and`, `or` and `xor` operations at [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). -* FEATURE: add support for automatic conversion of [RFC3339 time](https://www.rfc-editor.org/rfc/rfc3339) and IPv4 addresses into numeric representation at [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). -* FEATURE: add ability to format numeric fields into string representation of time, duration and IPv4 with [`format` pipe](./VictoriaLogs/LogsQL.md#format-pipe). -* FEATURE: set `format` field to `rfc3164` or `rfc5424` depending on the [Syslog format](https://en.wikipedia.org/wiki/Syslog) parsed via [`unpack_syslog` pipe](./VictoriaLogs/LogsQL.md#unpack_syslog-pipe). +* FEATURE: add [`pack_logfmt` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#pack_logfmt-pipe) for formatting [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into [logfmt](https://brandur.org/logfmt) messages. +* FEATURE: allow using IPv4 addresses in [range comparison filters](https://docs.victoriametrics.com/victorialogs/logsql/#range-comparison-filter). For example, `ip:>'12.34.56.78'` is valid filter now. +* FEATURE: add `ceil()` and `floor()` functions to [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). +* FEATURE: add support for bitwise `and`, `or` and `xor` operations at [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). +* FEATURE: add support for automatic conversion of [RFC3339 time](https://www.rfc-editor.org/rfc/rfc3339) and IPv4 addresses into numeric representation at [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). +* FEATURE: add ability to format numeric fields into string representation of time, duration and IPv4 with [`format` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe). +* FEATURE: set `format` field to `rfc3164` or `rfc5424` depending on the [Syslog format](https://en.wikipedia.org/wiki/Syslog) parsed via [`unpack_syslog` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_syslog-pipe). -* BUGFIX: always respect the limit set in [`limit` pipe](./VictoriaLogs/LogsQL.md#limit-pipe). Previously the limit could be exceeded in some cases. +* BUGFIX: always respect the limit set in [`limit` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe). Previously the limit could be exceeded in some cases. ## [v0.16.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.16.0-victorialogs) Released at 2024-06-04 -* FEATURE: add [`unpack_syslog` pipe](./VictoriaLogs/LogsQL.md#unpack_syslog-pipe) for unpacking [syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: parse timestamps in [`_time` filter](./VictoriaLogs/LogsQL.md#time-filter) with nanosecond precision. -* FEATURE: return the last `N` matching logs from [`/select/logsql/query` HTTP API](./VictoriaLogs/querying/README.md#querying-logs) with the maximum timestamps if `limit=N` query arg is passed to it. Previously a random subset of matching logs could be returned, which could complicate investigation of the returned logs. -* FEATURE: add [`drop_empty_fields` pipe](./VictoriaLogs/LogsQL.md#drop_empty_fields-pipe) for dropping [log fields](./VictoriaLogs/keyConcepts.md#data-model) with empty values. +* FEATURE: add [`unpack_syslog` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_syslog-pipe) for unpacking [syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: parse timestamps in [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) with nanosecond precision. +* FEATURE: return the last `N` matching logs from [`/select/logsql/query` HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-logs) with the maximum timestamps if `limit=N` query arg is passed to it. Previously a random subset of matching logs could be returned, which could complicate investigation of the returned logs. +* FEATURE: add [`drop_empty_fields` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#drop_empty_fields-pipe) for dropping [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with empty values. ## [v0.15.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.15.0-victorialogs) Released at 2024-05-30 -* FEATURE: add [`row_any`](./VictoriaLogs/LogsQL.md#row_any-stats) function for [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe). This function returns a sample log entry per every calculated [group of results](./VictoriaLogs/LogsQL.md#stats-by-fields). -* FEATURE: add `default` operator to [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). It allows overriding `NaN` results with the given default value. -* FEATURE: add `exp()` and `ln()` functions to [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe). -* FEATURE: allow omitting result name in [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe) expresions. In this case the result name is automatically set to string representation of the corresponding math expression. For example, `_time:5m | math duration / 1000` is equivalent to `_time:5m | math (duration / 1000) as "duration / 1000"`. -* FEATURE: allow omitting result name in [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe). In this case the result name is automatically set to string representation of the corresponding [stats function expression](./VictoriaLogs/LogsQL.md#stats-pipe-functions). For example, `_time:5m | count(*)` is valid [LogsQL query](./VictoriaLogs/LogsQL.md) now. It is equivalent to `_time:5m | stats count(*) as "count(*)"`. +* FEATURE: add [`row_any`](https://docs.victoriametrics.com/victorialogs/logsql/#row_any-stats) function for [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). This function returns a sample log entry per every calculated [group of results](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-fields). +* FEATURE: add `default` operator to [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). It allows overriding `NaN` results with the given default value. +* FEATURE: add `exp()` and `ln()` functions to [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe). +* FEATURE: allow omitting result name in [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe) expresions. In this case the result name is automatically set to string representation of the corresponding math expression. For example, `_time:5m | math duration / 1000` is equivalent to `_time:5m | math (duration / 1000) as "duration / 1000"`. +* FEATURE: allow omitting result name in [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). In this case the result name is automatically set to string representation of the corresponding [stats function expression](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions). For example, `_time:5m | count(*)` is valid [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/) now. It is equivalent to `_time:5m | stats count(*) as "count(*)"`. * BUGFIX: properly calculate the number of matching rows in `* | field_values x | stats count() rows` and in `* | unroll (x) | stats count() rows` queries. @@ -175,185 +175,185 @@ Released at 2024-05-30 Released at 2024-05-29 -* FEATURE: allow specifying fields, which must be packed into JSON in [`pack_json` pipe](./VictoriaLogs/LogsQL.md#pack_json-pipe) via `pack_json fields (field1, ..., fieldN)` syntax. +* FEATURE: allow specifying fields, which must be packed into JSON in [`pack_json` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#pack_json-pipe) via `pack_json fields (field1, ..., fieldN)` syntax. -* BUGFIX: properly apply `if (...)` filters to calculated results in [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe) when [grouping by fields](./VictoriaLogs/LogsQL.md#stats-by-fields) is enabled. For example, `_time:5m | stats by (host) count() logs, count() if (error) errors` now properly calculates per-`host` `errors`. +* BUGFIX: properly apply `if (...)` filters to calculated results in [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) when [grouping by fields](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-fields) is enabled. For example, `_time:5m | stats by (host) count() logs, count() if (error) errors` now properly calculates per-`host` `errors`. ## [v0.13.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.13.0-victorialogs) Released at 2024-05-28 -* FEATURE: add [`extract_regexp` pipe](./VictoriaLogs/LogsQL.md#extract_regexp-pipe) for extracting arbitrary substrings from [log fields](./VictoriaLogs/keyConcepts.md#data-model) with [RE2 regular expressions](https://github.com/google/re2/wiki/Syntax). -* FEATURE: add [`math` pipe](./VictoriaLogs/LogsQL.md#math-pipe) for mathematical calculations over [log fields](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: add [`field_values` pipe](./VictoriaLogs/LogsQL.md#field_values-pipe), which returns unique values for the given [log field](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: allow omitting `stats` prefix in [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe). For example, `_time:5m | count() rows` is a valid query now. It is equivalent to `_time:5m | stats count() as rows`. -* FEATURE: allow omitting `filter` prefix in [`filter` pipe](./VictoriaLogs/LogsQL.md#filter-pipe) if the filter doesn't clash with [pipe names](#./VictoriaLogs/LogsQL.md#pipes). For example, `_time:5m | stats by (host) count() rows | rows:>1000` is a valid query now. It is equivalent to `_time:5m | stats by (host) count() rows | filter rows:>1000`. -* FEATURE: allow [`head` pipe](./VictoriaLogs/LogsQL.md#limit-pipe) without number. For example, `error | head`. In this case 10 first values are returned as `head` Unix command does by default. -* FEATURE: allow using [comparison filters](./VictoriaLogs/LogsQL.md#range-comparison-filter) with strings. For example, `some_text_field:>="foo"` matches [log entries](./VictoriaLogs/keyConcepts.md#data-model) with `some_text_field` field values bigger or equal to `foo`. +* FEATURE: add [`extract_regexp` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#extract_regexp-pipe) for extracting arbitrary substrings from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with [RE2 regular expressions](https://github.com/google/re2/wiki/Syntax). +* FEATURE: add [`math` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#math-pipe) for mathematical calculations over [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: add [`field_values` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#field_values-pipe), which returns unique values for the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: allow omitting `stats` prefix in [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). For example, `_time:5m | count() rows` is a valid query now. It is equivalent to `_time:5m | stats count() as rows`. +* FEATURE: allow omitting `filter` prefix in [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe) if the filter doesn't clash with [pipe names](#https://docs.victoriametrics.com/victorialogs/logsql/#pipes). For example, `_time:5m | stats by (host) count() rows | rows:>1000` is a valid query now. It is equivalent to `_time:5m | stats by (host) count() rows | filter rows:>1000`. +* FEATURE: allow [`head` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe) without number. For example, `error | head`. In this case 10 first values are returned as `head` Unix command does by default. +* FEATURE: allow using [comparison filters](https://docs.victoriametrics.com/victorialogs/logsql/#range-comparison-filter) with strings. For example, `some_text_field:>="foo"` matches [log entries](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with `some_text_field` field values bigger or equal to `foo`. ## [v0.12.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.1-victorialogs) Released at 2024-05-26 -* FEATURE: add support for comments in multi-line LogsQL queries. See [these docs](./VictoriaLogs/LogsQL.md#comments). +* FEATURE: add support for comments in multi-line LogsQL queries. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#comments). -* BUGFIX: properly apply [`in(...)` filter](./VictoriaLogs/LogsQL.md#multi-exact-filter) inside `if (...)` conditions at various [pipes](./VictoriaLogs/LogsQL.md#pipes). This bug has been introduced in [v0.12.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.0-victorialogs). +* BUGFIX: properly apply [`in(...)` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter) inside `if (...)` conditions at various [pipes](https://docs.victoriametrics.com/victorialogs/logsql/#pipes). This bug has been introduced in [v0.12.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.0-victorialogs). ## [v0.12.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.0-victorialogs) Released at 2024-05-26 -* FEATURE: add [`pack_json` pipe](./VictoriaLogs/LogsQL.md#pack_json-pipe), which packs all the [log fields](./VictoriaLogs/keyConcepts.md#data-model) into a JSON object and stores it into the given field. -* FEATURE: add [`unroll` pipe](./VictoriaLogs/LogsQL.md#unroll-pipe), which can be used for unrolling JSON arrays stored in [log fields](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: add [`replace_regexp` pipe](./VictoriaLogs/LogsQL.md#replace_regexp-pipe), which allows updating [log fields](./VictoriaLogs/keyConcepts.md#data-model) with regular expressions. -* FEATURE: improve performance for [`format`](./VictoriaLogs/LogsQL.md#format-pipe) and [`extract`](./VictoriaLogs/LogsQL.md#extract-pipe) pipes. -* FEATURE: improve performance for [`/select/logsql/field_names` HTTP API](./VictoriaLogs/querying/README.md#querying-field-names). +* FEATURE: add [`pack_json` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#pack_json-pipe), which packs all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into a JSON object and stores it into the given field. +* FEATURE: add [`unroll` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unroll-pipe), which can be used for unrolling JSON arrays stored in [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: add [`replace_regexp` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#replace_regexp-pipe), which allows updating [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with regular expressions. +* FEATURE: improve performance for [`format`](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe) and [`extract`](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe) pipes. +* FEATURE: improve performance for [`/select/logsql/field_names` HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-field-names). -* BUGFIX: prevent from panic in [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe) when VictoriaLogs runs on a system with one CPU core. +* BUGFIX: prevent from panic in [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) when VictoriaLogs runs on a system with one CPU core. * BUGFIX: do not return referenced fields if they weren't present in the original logs. For example, `_time:5m | format if (non_existing_field:"") "abc"` could return empty `non_exiting_field`, while it shouldn't be returned because it is missing in the original logs. -* BUGFIX: properly initialize values for [`in(...)` filter](./VictoriaLogs/LogsQL.md#multi-exact-filter) inside [`filter` pipe](./VictoriaLogs/LogsQL.md#filter-pipe) if the `in(...)` contains other [filters](./VictoriaLogs/LogsQL.md#filters). For example, `_time:5m | filter ip:in(user_type:admin | fields ip)` now works correctly. +* BUGFIX: properly initialize values for [`in(...)` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter) inside [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe) if the `in(...)` contains other [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters). For example, `_time:5m | filter ip:in(user_type:admin | fields ip)` now works correctly. ## [v0.11.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.11.0-victorialogs) Released at 2024-05-25 -* FEATURE: add [`replace` pipe](./VictoriaLogs/LogsQL.md#replace-pipe), which allows replacing substrings in [log fields](./VictoriaLogs/keyConcepts.md#data-model). -* FEATURE: support [comparing](./VictoriaLogs/LogsQL.md#range-filter) log field values with [special numeric values](./VictoriaLogs/LogsQL.md#numeric-values). For example, `duration:>1.5s` and `response_size:<15KiB` are valid filters now. -* FEATURE: properly sort [durations](./VictoriaLogs/LogsQL.md#duration-values) and [short numeric values](./VictoriaLogs/LogsQL.md#short-numeric-values) in [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe). For example, `10s` goes in front of `1h`, while `10KB` goes in front of `1GB`. -* FEATURE: add an ability to preserve the original non-empty field values when executing [`extract`](./VictoriaLogs/LogsQL.md#extract-pipe), [`unpack_json`](./VictoriaLogs/LogsQL.md#unpack_json-pipe), [`unpack_logfmt`](./VictoriaLogs/LogsQL.md#unpack_logfmt-pipe) and [`format`](./VictoriaLogs/LogsQL.md#format-pipe) pipes. -* FEATURE: add an ability to preserve the original field values if the corresponding unpacked values are empty when executing [`extract`](./VictoriaLogs/LogsQL.md#extract-pipe), [`unpack_json`](./VictoriaLogs/LogsQL.md#unpack_json-pipe), [`unpack_logfmt`](./VictoriaLogs/LogsQL.md#unpack_logfmt-pipe) and [`format`](./VictoriaLogs/LogsQL.md#format-pipe) pipes. +* FEATURE: add [`replace` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#replace-pipe), which allows replacing substrings in [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +* FEATURE: support [comparing](https://docs.victoriametrics.com/victorialogs/logsql/#range-filter) log field values with [special numeric values](https://docs.victoriametrics.com/victorialogs/logsql/#numeric-values). For example, `duration:>1.5s` and `response_size:<15KiB` are valid filters now. +* FEATURE: properly sort [durations](https://docs.victoriametrics.com/victorialogs/logsql/#duration-values) and [short numeric values](https://docs.victoriametrics.com/victorialogs/logsql/#short-numeric-values) in [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). For example, `10s` goes in front of `1h`, while `10KB` goes in front of `1GB`. +* FEATURE: add an ability to preserve the original non-empty field values when executing [`extract`](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe), [`unpack_json`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe), [`unpack_logfmt`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_logfmt-pipe) and [`format`](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe) pipes. +* FEATURE: add an ability to preserve the original field values if the corresponding unpacked values are empty when executing [`extract`](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe), [`unpack_json`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe), [`unpack_logfmt`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_logfmt-pipe) and [`format`](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe) pipes. ## [v0.10.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.10.0-victorialogs) Released at 2024-05-24 -* FEATURE: return the number of matching log entries per returned value in [HTTP API](./VictoriaLogs/querying/README.md#http-api) results. This simplifies detecting [field](./VictoriaLogs/keyConcepts.md#data-model) / [stream](./VictoriaLogs/keyConcepts.md#stream-fields) values with the biggest number of logs for the given [LogsQL query](./VictoriaLogs/LogsQL.md). -* FEATURE: improve performance for [regexp filter](./VictoriaLogs/LogsQL.md#regexp-filter) in the following cases: +* FEATURE: return the number of matching log entries per returned value in [HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#http-api) results. This simplifies detecting [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) / [stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) values with the biggest number of logs for the given [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/). +* FEATURE: improve performance for [regexp filter](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) in the following cases: - If the regexp contains just a phrase without special regular expression chars. For example, `~"foo"`. - If the regexp starts with `.*` or ends with `.*`. For example, `~".*foo.*"`. - If the regexp contains multiple strings delimited by `|`. For example, `~"foo|bar|baz"`. - - If the regexp contains multiple [words](./VictoriaLogs/LogsQL.md#word). For example, `~"foo bar baz"`. -* FEATURE: allow disabling automatic unquoting of the matched placeholders in [`extract` pipe](./VictoriaLogs/LogsQL.md#extract-pipe). See [these docs](./VictoriaLogs/LogsQL.md#format-for-extract-pipe-pattern). + - If the regexp contains multiple [words](https://docs.victoriametrics.com/victorialogs/logsql/#word). For example, `~"foo bar baz"`. +* FEATURE: allow disabling automatic unquoting of the matched placeholders in [`extract` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#format-for-extract-pipe-pattern). -* BUGFIX: properly parse `!` in front of [exact filter](./VictoriaLogs/LogsQL.md#exact-filter), [exact-prefix filter](./VictoriaLogs/LogsQL.md#exact-prefix-filter) and [regexp filter](./VictoriaLogs/LogsQL.md#regexp-filter). For example, `!~"some regexp"` is properly parsed as `not ="some regexp"`. Previously it was incorrectly parsed as `'~="some regexp"'` [phrase filter](./VictoriaLogs/LogsQL.md#phrase-filter). -* BUGFIX: properly sort results by [`_time` field](./VictoriaLogs/keyConcepts.md#time-field) when [`limit` pipe](./VictoriaLogs/LogsQL.md#limit-pipe) is applied. For example, `_time:5m | sort by (_time) desc | limit 10` properly works now. +* BUGFIX: properly parse `!` in front of [exact filter](https://docs.victoriametrics.com/victorialogs/logsql/#exact-filter), [exact-prefix filter](https://docs.victoriametrics.com/victorialogs/logsql/#exact-prefix-filter) and [regexp filter](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter). For example, `!~"some regexp"` is properly parsed as `not ="some regexp"`. Previously it was incorrectly parsed as `'~="some regexp"'` [phrase filter](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter). +* BUGFIX: properly sort results by [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) when [`limit` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe) is applied. For example, `_time:5m | sort by (_time) desc | limit 10` properly works now. ## [v0.9.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.1-victorialogs) Released at 2024-05-22 -* BUGFIX: [web UI](./VictoriaLogs/querying/README.md#web-ui): fix loading web UI, which has been broken in [v0.9.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.0-victorialogs). +* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): fix loading web UI, which has been broken in [v0.9.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.0-victorialogs). ## [v0.9.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.9.0-victorialogs) Released at 2024-05-22 -* FEATURE: allow using `~"some_regexp"` [regexp filter](./VictoriaLogs/LogsQL.md#regexp-filter) instead of `re("some_regexp")`. -* FEATURE: allow using `="some phrase"` [exact filter](./VictoriaLogs/LogsQL.md#exact-filter) instead of `exact("some phrase")`. -* FEATURE: allow using `="some prefix"*` [exact prefix filter](./VictoriaLogs/LogsQL.md#exact-prefix-filter) instead of `exact("some prefix"*)`. -* FEATURE: add ability to generate output fields according to the provided format string. See [these docs](./VictoriaLogs/LogsQL.md#format-pipe). -* FEATURE: add ability to extract fields with [`extract` pipe](./VictoriaLogs/LogsQL.md#extract-pipe) only if the given condition is met. See [these docs](./VictoriaLogs/LogsQL.md#conditional-extract). -* FEATURE: add ability to unpack JSON fields with [`unpack_json` pipe](./VictoriaLogs/LogsQL.md#unpack_json-pipe) only if the given condition is met. See [these docs](./VictoriaLogs/LogsQL.md#conditional-unpack_json). -* FEATURE: add ability to unpack [logfmt](https://brandur.org/logfmt) fields with [`unpack_logfmt` pipe](./VictoriaLogs/LogsQL.md#unpack_logfmt-pipe) only if the given condition is met. See [these docs](./VictoriaLogs/LogsQL.md#conditional-unpack_logfmt). -* FEATURE: add [`row_min`](./VictoriaLogs/LogsQL.md#row_min-stats) and [`row_max`](./VictoriaLogs/LogsQL.md#row_max-stats) functions for [`stats` pipe](./VictoriaLogs/LogsQL.md#stats-pipe), which allow returning all the [log fields](./VictoriaLogs/keyConcepts.md#data-model) for the log entry with the minimum / maximum value at the given field. -* FEATURE: add `/select/logsql/streams` HTTP endpoint for returning [streams](./VictoriaLogs/keyConcepts.md#stream-fields) from results of the given query. See [these docs](./VictoriaLogs/querying/README.md#querying-streams) for details. -* FEATURE: add `/select/logsql/stream_field_names` HTTP endpoint for returning [stream](./VictoriaLogs/keyConcepts.md#stream-fields) field names from results of the given query. See [these docs](./VictoriaLogs/querying/README.md#querying-stream-field-names) for details. -* FEATURE: add `/select/logsql/stream_field_values` HTTP endpoint for returning [stream](./VictoriaLogs/keyConcepts.md#stream-fields) field values for the given label from results of the given query. See [these docs](./VictoriaLogs/querying/README.md#querying-stream-field-values) for details. -* FEATURE: [web UI](./VictoriaLogs/querying/README.md#web-ui): change time range limitation from `_time` in the expression to `start` and `end` query args. +* FEATURE: allow using `~"some_regexp"` [regexp filter](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) instead of `re("some_regexp")`. +* FEATURE: allow using `="some phrase"` [exact filter](https://docs.victoriametrics.com/victorialogs/logsql/#exact-filter) instead of `exact("some phrase")`. +* FEATURE: allow using `="some prefix"*` [exact prefix filter](https://docs.victoriametrics.com/victorialogs/logsql/#exact-prefix-filter) instead of `exact("some prefix"*)`. +* FEATURE: add ability to generate output fields according to the provided format string. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#format-pipe). +* FEATURE: add ability to extract fields with [`extract` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe) only if the given condition is met. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#conditional-extract). +* FEATURE: add ability to unpack JSON fields with [`unpack_json` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe) only if the given condition is met. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#conditional-unpack_json). +* FEATURE: add ability to unpack [logfmt](https://brandur.org/logfmt) fields with [`unpack_logfmt` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_logfmt-pipe) only if the given condition is met. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#conditional-unpack_logfmt). +* FEATURE: add [`row_min`](https://docs.victoriametrics.com/victorialogs/logsql/#row_min-stats) and [`row_max`](https://docs.victoriametrics.com/victorialogs/logsql/#row_max-stats) functions for [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe), which allow returning all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) for the log entry with the minimum / maximum value at the given field. +* FEATURE: add `/select/logsql/streams` HTTP endpoint for returning [streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) from results of the given query. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-streams) for details. +* FEATURE: add `/select/logsql/stream_field_names` HTTP endpoint for returning [stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) field names from results of the given query. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-names) for details. +* FEATURE: add `/select/logsql/stream_field_values` HTTP endpoint for returning [stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) field values for the given label from results of the given query. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-values) for details. +* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): change time range limitation from `_time` in the expression to `start` and `end` query args. -* BUGFIX: fix `invalid memory address or nil pointer dereference` panic when using [`extract`](./VictoriaLogs/LogsQL.md#extract-pipe), [`unpack_json`](./VictoriaLogs/LogsQL.md#unpack_json-pipe) or [`unpack_logfmt`](./VictoriaLogs/LogsQL.md#unpack_logfmt-pipe) pipes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6306). -* BUGFIX: [web UI](./VictoriaLogs/querying/README.md#web-ui): fix an issue where logs with long `_msg` values might not display. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6281). +* BUGFIX: fix `invalid memory address or nil pointer dereference` panic when using [`extract`](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe), [`unpack_json`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe) or [`unpack_logfmt`](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_logfmt-pipe) pipes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6306). +* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): fix an issue where logs with long `_msg` values might not display. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6281). * BUGFIX: properly handle time range boundaries with millisecond precision. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6293). ## [v0.8.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.8.0-victorialogs) Released at 2024-05-20 -* FEATURE: add ability to extract JSON fields from [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](./VictoriaLogs/LogsQL.md#unpack_json-pipe). -* FEATURE: add ability to extract [logfmt](https://brandur.org/logfmt) fields from [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](./VictoriaLogs/LogsQL.md#unpack_logfmt-pipe). -* FEATURE: add ability to extract arbitrary text from [log fields](./VictoriaLogs/keyConcepts.md#data-model) into the output fields. See [these docs](./VictoriaLogs/LogsQL.md#extract-pipe). -* FEATURE: add ability to put arbitrary [queries](./VictoriaLogs/LogsQL.md#query-syntax) inside [`in()` filter](./VictoriaLogs/LogsQL.md#multi-exact-filter). -* FEATURE: add support for post-filtering of query results with [`filter` pipe](./VictoriaLogs/LogsQL.md#filter-pipe). -* FEATURE: allow applying individual [filters](./VictoriaLogs/LogsQL.md#filters) per each [stats function](./VictoriaLogs/LogsQL.md#stats-pipe-functions). See [these docs](./VictoriaLogs/LogsQL.md#stats-with-additional-filters). -* FEATURE: allow passing string values to [`min`](./VictoriaLogs/LogsQL.md#min-stats) and [`max`](./VictoriaLogs/LogsQL.md#max-stats) functions. Previously only numeric values could be passed to them. -* FEATURE: speed up [`sort ... limit N` pipe](./VictoriaLogs/LogsQL.md#sort-pipe) for typical cases. -* FEATURE: allow using more convenient syntax for [`range` filters](./VictoriaLogs/LogsQL.md#range-filter) if upper or lower bound isn't needed. For example, it is possible to write `response_size:>=10KiB` instead of `response_size:range[10KiB, inf)`, or `temperature:<42` instead of `temperature:range(-inf, 42)`. -* FEATURE: add `/select/logsql/hits` HTTP endpoint for returning the number of matching logs per the given time bucket over the selected time range. See [these docs](./VictoriaLogs/querying/README.md#querying-hits-stats) for details. -* FEATURE: add `/select/logsql/field_names` HTTP endpoint for returning [field](./VictoriaLogs/keyConcepts.md#data-model) names from results of the given query. See [these docs](./VictoriaLogs/querying/README.md#querying-field-names) for details. -* FEATURE: add `/select/logsql/field_values` HTTP endpoint for returning unique values for the given [field](./VictoriaLogs/keyConcepts.md#data-model) obtained from results of the given query. See [these docs](./VictoriaLogs/querying/README.md#querying-field-values) for details. +* FEATURE: add ability to extract JSON fields from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe). +* FEATURE: add ability to extract [logfmt](https://brandur.org/logfmt) fields from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_logfmt-pipe). +* FEATURE: add ability to extract arbitrary text from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into the output fields. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe). +* FEATURE: add ability to put arbitrary [queries](https://docs.victoriametrics.com/victorialogs/logsql/#query-syntax) inside [`in()` filter](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter). +* FEATURE: add support for post-filtering of query results with [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe). +* FEATURE: allow applying individual [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) per each [stats function](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#stats-with-additional-filters). +* FEATURE: allow passing string values to [`min`](https://docs.victoriametrics.com/victorialogs/logsql/#min-stats) and [`max`](https://docs.victoriametrics.com/victorialogs/logsql/#max-stats) functions. Previously only numeric values could be passed to them. +* FEATURE: speed up [`sort ... limit N` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) for typical cases. +* FEATURE: allow using more convenient syntax for [`range` filters](https://docs.victoriametrics.com/victorialogs/logsql/#range-filter) if upper or lower bound isn't needed. For example, it is possible to write `response_size:>=10KiB` instead of `response_size:range[10KiB, inf)`, or `temperature:<42` instead of `temperature:range(-inf, 42)`. +* FEATURE: add `/select/logsql/hits` HTTP endpoint for returning the number of matching logs per the given time bucket over the selected time range. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-hits-stats) for details. +* FEATURE: add `/select/logsql/field_names` HTTP endpoint for returning [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) names from results of the given query. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-field-names) for details. +* FEATURE: add `/select/logsql/field_values` HTTP endpoint for returning unique values for the given [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) obtained from results of the given query. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#querying-field-values) for details. -* BUGFIX: properly take into account `offset` at [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe) when it already has `limit`. For example, `_time:5m | sort by (foo) offset 20 limit 10`. +* BUGFIX: properly take into account `offset` at [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) when it already has `limit`. For example, `_time:5m | sort by (foo) offset 20 limit 10`. ## [v0.7.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.7.0-victorialogs) Released at 2024-05-15 -* FEATURE: add support for optional `start` and `end` query args to [HTTP querying API](./VictoriaLogs/querying/README.md#http-api), which can be used for limiting the time range for [LogsQL query](./VictoriaLogs/LogsQL.md). -* FEATURE: add ability to return the first `N` results from [`sort` pipe](#./VictoriaLogs/LogsQL.md#sort-pipe). This is useful when `N` biggest or `N` smallest values must be returned from large amounts of logs. -* FEATURE: add [`quantile`](./VictoriaLogs/LogsQL.md#quantile-stats) and [`median`](./VictoriaLogs/LogsQL.md#median-stats) [stats functions](./VictoriaLogs/LogsQL.md#stats-pipe). +* FEATURE: add support for optional `start` and `end` query args to [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api), which can be used for limiting the time range for [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/). +* FEATURE: add ability to return the first `N` results from [`sort` pipe](#https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). This is useful when `N` biggest or `N` smallest values must be returned from large amounts of logs. +* FEATURE: add [`quantile`](https://docs.victoriametrics.com/victorialogs/logsql/#quantile-stats) and [`median`](https://docs.victoriametrics.com/victorialogs/logsql/#median-stats) [stats functions](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). ## [v0.6.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.1-victorialogs) Released at 2024-05-14 -* FEATURE: use [natural sort order](https://en.wikipedia.org/wiki/Natural_sort_order) when sorting logs via [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe). +* FEATURE: use [natural sort order](https://en.wikipedia.org/wiki/Natural_sort_order) when sorting logs via [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). -* BUGFIX: properly return matching logs in [streams](./VictoriaLogs/keyConcepts.md#stream-fields) with small number of entries. Previously they could be skipped. The issue has been introduced in [the release v0.6.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.0-victorialogs). -* BUGFIX: fix `runtime error: index out of range` panic when using [`sort` pipe](./VictoriaLogs/LogsQL.md#sort-pipe) like `_time:1h | sort by (_time)`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258). +* BUGFIX: properly return matching logs in [streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with small number of entries. Previously they could be skipped. The issue has been introduced in [the release v0.6.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.0-victorialogs). +* BUGFIX: fix `runtime error: index out of range` panic when using [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) like `_time:1h | sort by (_time)`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258). ## [v0.6.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.0-victorialogs) Released at 2024-05-12 -* FEATURE: return all the log fields by default in query results. Previously only [`_stream`](./VictoriaLogs/keyConcepts.md#stream-fields), [`_time`](./VictoriaLogs/keyConcepts.md#time-field) and [`_msg`](./VictoriaLogs/keyConcepts.md#message-field) fields were returned by default. -* FEATURE: add support for returning only the requested log [fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](./VictoriaLogs/LogsQL.md#fields-pipe). -* FEATURE: add support for calculating various stats over [log fields](./VictoriaLogs/keyConcepts.md#data-model). Grouping by arbitrary set of [log fields](./VictoriaLogs/keyConcepts.md#data-model) is supported. See [these docs](./VictoriaLogs/LogsQL.md#stats-pipe) for details. -* FEATURE: add support for sorting the returned results. See [these docs](./VictoriaLogs/LogsQL.md#sort-pipe). -* FEATURE: add support for returning unique results. See [these docs](./VictoriaLogs/LogsQL.md#uniq-pipe). -* FEATURE: add support for limiting the number of returned results. See [these docs](./VictoriaLogs/LogsQL.md#limiters). -* FEATURE: add support for copying and renaming the selected log fields. See [these](./VictoriaLogs/LogsQL.md#copy-pipe) and [these](./VictoriaLogs/LogsQL.md#rename-pipe) docs. -* FEATURE: allow using `_` inside numbers. For example, `score:range[1_000, 5_000_000]` for [`range` filter](./VictoriaLogs/LogsQL.md#range-filter). -* FEATURE: allow numbers in hexadecimal and binary form. For example, `response_size:range[0xff, 0b10001101101]` for [`range` filter](./VictoriaLogs/LogsQL.md#range-filter). -* FEATURE: allow using duration and byte size suffixes in numeric values inside LogsQL queries. See [these docs](./VictoriaLogs/LogsQL.md#numeric-values). +* FEATURE: return all the log fields by default in query results. Previously only [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields), [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) and [`_msg`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) fields were returned by default. +* FEATURE: add support for returning only the requested log [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe). +* FEATURE: add support for calculating various stats over [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). Grouping by arbitrary set of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) is supported. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) for details. +* FEATURE: add support for sorting the returned results. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). +* FEATURE: add support for returning unique results. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#uniq-pipe). +* FEATURE: add support for limiting the number of returned results. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#limiters). +* FEATURE: add support for copying and renaming the selected log fields. See [these](https://docs.victoriametrics.com/victorialogs/logsql/#copy-pipe) and [these](https://docs.victoriametrics.com/victorialogs/logsql/#rename-pipe) docs. +* FEATURE: allow using `_` inside numbers. For example, `score:range[1_000, 5_000_000]` for [`range` filter](https://docs.victoriametrics.com/victorialogs/logsql/#range-filter). +* FEATURE: allow numbers in hexadecimal and binary form. For example, `response_size:range[0xff, 0b10001101101]` for [`range` filter](https://docs.victoriametrics.com/victorialogs/logsql/#range-filter). +* FEATURE: allow using duration and byte size suffixes in numeric values inside LogsQL queries. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#numeric-values). * FEATURE: improve data ingestion performance by up to 50%. -* FEATURE: optimize performance for [LogsQL query](./VictoriaLogs/LogsQL.md), which contains multiple filters for [words](./VictoriaLogs/LogsQL.md#word-filter) or [phrases](./VictoriaLogs/LogsQL.md#phrase-filter) delimited with [`AND` operator](./VictoriaLogs/LogsQL.md#logical-filter). For example, `foo AND bar` query must find [log messages](./VictoriaLogs/keyConcepts.md#message-field) with `foo` and `bar` words at faster speed. +* FEATURE: optimize performance for [LogsQL query](https://docs.victoriametrics.com/victorialogs/logsql/), which contains multiple filters for [words](https://docs.victoriametrics.com/victorialogs/logsql/#word-filter) or [phrases](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter) delimited with [`AND` operator](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter). For example, `foo AND bar` query must find [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with `foo` and `bar` words at faster speed. -* BUGFIX: prevent from possible corruption of short [log fields](./VictoriaLogs/keyConcepts.md#data-model) during data ingestion. +* BUGFIX: prevent from possible corruption of short [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) during data ingestion. * BUGFIX: prevent from additional CPU usage for up to a few seconds after canceling the query. -* BUGFIX: prevent from returning log entries with emtpy `_stream` field in the form `"_stream":""` in [search query results](./VictoriaLogs/querying/README.md). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6042). +* BUGFIX: prevent from returning log entries with emtpy `_stream` field in the form `"_stream":""` in [search query results](https://docs.victoriametrics.com/victorialogs/querying/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6042). ## [v0.5.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.2-victorialogs) Released at 2024-04-11 -* BUGFIX: properly register new [log streams](./VictoriaLogs/keyConcepts.md#stream-fields) under high data ingestion rate. The issue has been introduced in [v0.5.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs). +* BUGFIX: properly register new [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) under high data ingestion rate. The issue has been introduced in [v0.5.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs). ## [v0.5.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.1-victorialogs) Released at 2024-04-04 -* BUGFIX: properly apply time range filter for queries containing [`OR` operators](./VictoriaLogs/LogsQL.md#logical-filter). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920). +* BUGFIX: properly apply time range filter for queries containing [`OR` operators](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920). * BUGFIX: do not log debug lines `DEBUG: start trimLines` and `DEBUG: end trimLines`. This bug has been introduced in [v0.5.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs) in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/0514091948cf8e00e42f44318c0e5e5b63b6388f). ## [v0.5.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs) Released at 2024-03-01 -* FEATURE: support the ability to limit the number of returned log entries from [HTTP querying API](./VictoriaLogs/querying/README.md#http-api) by passing `limit` query arg. Previously all the matching log entries were returned until closing the response stream. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674). Thanks to @dmitryk-dk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778). +* FEATURE: support the ability to limit the number of returned log entries from [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api) by passing `limit` query arg. Previously all the matching log entries were returned until closing the response stream. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674). Thanks to @dmitryk-dk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778). -* BUGFIX: do not panic on incorrect regular expression in [stream filter](./VictoriaLogs/LogsQL.md#stream-filter). Thanks to @XLONG96 for [the bugfix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5897). +* BUGFIX: do not panic on incorrect regular expression in [stream filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter). Thanks to @XLONG96 for [the bugfix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5897). * BUGFIX: properly determine when the assisted merge is needed. Previously the logs for determining whether the assisted merge is needed was broken. This could lead to too big number of parts under high data ingestion rate. Thanks to @lujiajing1126 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5447). -* BUGFIX: properly stop execution of aborted query when the query doesn't contain [`_stream` filter](./VictoriaLogs/LogsQL.md#stream-filter). Previously such a query could continue consuming resources after being aborted by the client. Thanks to @z-anshun for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5400). +* BUGFIX: properly stop execution of aborted query when the query doesn't contain [`_stream` filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter). Previously such a query could continue consuming resources after being aborted by the client. Thanks to @z-anshun for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5400). ## [v0.4.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.4.2-victorialogs) Released at 2023-11-15 -* BUGFIX: properly locate logs for the [requested streams](./VictoriaLogs/LogsQL.md#stream-filter). Previously logs for some streams may be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856). Thanks to @XLONG96 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295)! -* BUGFIX: [web UI](./VictoriaLogs/querying/README.md#web-ui): properly sort found logs by time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300). +* BUGFIX: properly locate logs for the [requested streams](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter). Previously logs for some streams may be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856). Thanks to @XLONG96 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295)! +* BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): properly sort found logs by time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300). ## [v0.4.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.4.1-victorialogs) @@ -365,12 +365,12 @@ Released at 2023-10-04 Released at 2023-10-03 -* FEATURE: add `-elasticsearch.version` command-line flag, which can be used for specifying Elasticsearch version returned by VictoriaLogs to Filebeat at [elasticsearch bulk API](./VictoriaLogs/data-ingestion/README.md#elasticsearch-bulk-api). This helps resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777). -* FEATURE: expose the following metrics at [/metrics](./VictoriaLogs/README.md#monitoring) page: - * `vl_data_size_bytes{type="storage"}` - on-disk size for data excluding [log stream](./VictoriaLogs/keyConcepts.md#stream-fields) indexes. - * `vl_data_size_bytes{type="indexdb"}` - on-disk size for [log stream](./VictoriaLogs/keyConcepts.md#stream-fields) indexes. +* FEATURE: add `-elasticsearch.version` command-line flag, which can be used for specifying Elasticsearch version returned by VictoriaLogs to Filebeat at [elasticsearch bulk API](https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api). This helps resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777). +* FEATURE: expose the following metrics at [/metrics](https://docs.victoriametrics.com/victorialogs/#monitoring) page: + * `vl_data_size_bytes{type="storage"}` - on-disk size for data excluding [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) indexes. + * `vl_data_size_bytes{type="indexdb"}` - on-disk size for [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) indexes. * FEATURE: add `-insert.maxFieldsPerLine` command-line flag, which can be used for limiting the number of fields per line in logs sent to VictoriaLogs via ingestion protocols. This helps to avoid issues like [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762). -* FEATURE: expose `vl_http_request_duration_seconds` histogram at the [/metrics](./VictoriaLogs/README.md#monitoring) page. Thanks to @crossoverJie for [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4934). +* FEATURE: expose `vl_http_request_duration_seconds` histogram at the [/metrics](https://docs.victoriametrics.com/victorialogs/#monitoring) page. Thanks to @crossoverJie for [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4934). * FEATURE: add support of `-storage.minFreeDiskSpaceBytes` command-line flag to allow switching to read-only mode when running out of disk space at `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737). * BUGFIX: fix possible panic when no data is written to VictoriaLogs for a long time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4895). Thanks to @crossoverJie for filing and fixing the issue. @@ -381,15 +381,15 @@ Released at 2023-10-03 Released at 2023-07-20 -* FEATURE: add support for data ingestion via Promtail (aka default log shipper for Grafana Loki). See [these](./VictoriaLogs/data-ingestion/Promtail.md) and [these](./VictoriaLogs/data-ingestion/README.md#loki-json-api) docs. +* FEATURE: add support for data ingestion via Promtail (aka default log shipper for Grafana Loki). See [these](https://docs.victoriametrics.com/victorialogs/data-ingestion/promtail/) and [these](https://docs.victoriametrics.com/victorialogs/data-ingestion/#loki-json-api) docs. ## [v0.2.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.2.0-victorialogs) Released at 2023-07-17 -* FEATURE: support short form of `_time` filters over the last X minutes/hours/days/etc. For example, `_time:5m` is a short form for `_time:(now-5m, now]`, which matches logs with [timestamps](./VictoriaLogs/keyConcepts.md#time-field) for the last 5 minutes. See [these docs](./VictoriaLogs/LogsQL.md#time-filter) for details. -* FEATURE: add ability to specify offset for the selected time range. For example, `_time:5m offset 1h` is equivalent to `_time:(now-5m-1h, now-1h]`. See [these docs](./VictoriaLogs/LogsQL.md#time-filter) for details. -* FEATURE: [LogsQL](./VictoriaLogs/LogsQL.md): replace `exact_prefix("...")` with `exact("..."*)`. This makes it consistent with [i()](./VictoriaLogs/LogsQL.md#case-insensitive-filter) filter, which can accept phrases and prefixes, e.g. `i("phrase")` and `i("phrase"*)`. See [these docs](./VictoriaLogs/LogsQL.md#exact-prefix-filter). +* FEATURE: support short form of `_time` filters over the last X minutes/hours/days/etc. For example, `_time:5m` is a short form for `_time:(now-5m, now]`, which matches logs with [timestamps](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) for the last 5 minutes. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for details. +* FEATURE: add ability to specify offset for the selected time range. For example, `_time:5m offset 1h` is equivalent to `_time:(now-5m-1h, now-1h]`. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for details. +* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): replace `exact_prefix("...")` with `exact("..."*)`. This makes it consistent with [i()](https://docs.victoriametrics.com/victorialogs/logsql/#case-insensitive-filter) filter, which can accept phrases and prefixes, e.g. `i("phrase")` and `i("phrase"*)`. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#exact-prefix-filter). ## [v0.1.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.1.0-victorialogs) diff --git a/docs/VictoriaLogs/LogsQL.md b/docs/VictoriaLogs/LogsQL.md index 5c6a4e7b8..1cf4d3bdb 100644 --- a/docs/VictoriaLogs/LogsQL.md +++ b/docs/VictoriaLogs/LogsQL.md @@ -9,13 +9,13 @@ menu: aliases: - /VictoriaLogs/LogsQL.html --- -LogsQL is a simple yet powerful query language for [VictoriaLogs](./VictoriaLogs/README.md). -See [examples](./VictoriaLogs/logsql-examples.md) and [tutorial](#logsql-tutorial) +LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/). +See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [tutorial](#logsql-tutorial) in order to feel the language. LogsQL provides the following features: -- Full-text search across [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- Full-text search across [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [word filter](#word-filter), [phrase filter](#phrase-filter) and [prefix filter](#prefix-filter). - Ability to combine filters into arbitrary complex [logical filters](#logical-filter). - Ability to extract structured fields from unstructured logs at query time. See [these docs](#transformations). @@ -23,21 +23,21 @@ LogsQL provides the following features: ## LogsQL tutorial -If you aren't familiar with VictoriaLogs, then start with [key concepts docs](./VictoriaLogs/keyConcepts.md). +If you aren't familiar with VictoriaLogs, then start with [key concepts docs](https://docs.victoriametrics.com/victorialogs/keyconcepts/). Then follow these docs: -- [How to run VictoriaLogs](./VictoriaLogs/QuickStart.md). -- [how to ingest data into VictoriaLogs](./VictoriaLogs/data-ingestion/README.md). -- [How to query VictoriaLogs](./VictoriaLogs/querying/README.md +- [How to run VictoriaLogs](https://docs.victoriametrics.com/victorialogs/quickstart/). +- [how to ingest data into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/victorialogs/querying/). -The simplest LogsQL query is just a [word](#word), which must be found in the [log message](./VictoriaLogs/keyConcepts.md#message-field). +The simplest LogsQL query is just a [word](#word), which must be found in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). For example, the following query finds all the logs with `error` word: ```logsql error ``` -See [how to send queries to VictoriaLogs](./VictoriaLogs/querying/README.md). +See [how to send queries to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/querying/). If the queried [word](#word) clashes with LogsQL keywords, then just wrap it into quotes. For example, the following query finds all the log messages with `and` [word](#word): @@ -59,7 +59,7 @@ finds log messages with the `error: cannot find file` phrase: "error: cannot find file" ``` -Queries above match logs with any [timestamp](./VictoriaLogs/keyConcepts.md#time-field), +Queries above match logs with any [timestamp](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field), e.g. they may return logs from the previous year alongside recently ingested logs. Usually logs from the previous year aren't so interesting comparing to the recently ingested logs. @@ -74,9 +74,9 @@ error AND _time:5m This query consists of two [filters](#filters) joined with `AND` [operator](#logical-filter): - The filter on the `error` [word](#word). -- The filter on the [`_time` field](./VictoriaLogs/keyConcepts.md#time-field). +- The filter on the [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field). -The `AND` operator means that the [log entry](./VictoriaLogs/keyConcepts.md#data-model) must match both filters in order to be selected. +The `AND` operator means that the [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) must match both filters in order to be selected. Typical LogsQL query consists of multiple [filters](#filters) joined with `AND` operator. It may be tiresome typing and then reading all these `AND` words. So LogsQL allows omitting `AND` words. For example, the following query is equivalent to the query above: @@ -87,7 +87,7 @@ _time:5m error The query returns logs in arbitrary order because sorting of big amounts of logs may require non-trivial amounts of CPU and RAM. The number of logs with `error` word over the last 5 minutes isn't usually too big (e.g. less than a few millions), so it is OK to sort them with [`sort` pipe](#sort-pipe). -The following query sorts the selected logs by [`_time`](./VictoriaLogs/keyConcepts.md#time-field) field: +The following query sorts the selected logs by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) field: ```logsql _time:5m error | sort by (_time) @@ -100,10 +100,10 @@ with [`limit` pipe](#limit-pipe). The following query returns the last 10 logs w _time:5m error | sort by (_time) desc | limit 10 ``` -By default VictoriaLogs returns all the [log fields](./VictoriaLogs/keyConcepts.md#data-model). +By default VictoriaLogs returns all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). If you need only the given set of fields, then add [`fields` pipe](#fields-pipe) to the end of the query. For example, the following query returns only -[`_time`](./VictoriaLogs/keyConcepts.md#time-field), [`_stream`](./VictoriaLogs/keyConcepts.md#stream-fields) -and [`_msg`](./VictoriaLogs/keyConcepts.md#message-field) fields: +[`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field), [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) +and [`_msg`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) fields: ```logsql error _time:5m | fields _time, _stream, _msg @@ -117,8 +117,8 @@ _time:5m error NOT buggy_app ``` This query uses `NOT` [operator](#logical-filter) for removing log lines from the buggy app. The `NOT` operator is used frequently, so it can be substituted with `!` char -(the `!` char is used instead of `-` char as a shorthand for `NOT` operator because it nicely combines with [`=`](./VictoriaLogs/LogsQL.md#exact-filter) -and [`~`](./VictoriaLogs/LogsQL.md#regexp-filter) filters like `!=` and `!~`). +(the `!` char is used instead of `-` char as a shorthand for `NOT` operator because it nicely combines with [`=`](https://docs.victoriametrics.com/victorialogs/logsql/#exact-filter) +and [`~`](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) filters like `!=` and `!~`). The following query is equivalent to the previous one: ```logsql @@ -144,8 +144,8 @@ This query returns logs with `foobar` [word](#word), even if do not contain `err So it is recommended wrapping the needed query parts into explicit parentheses if you are unsure in priority rules. As an additional bonus, explicit parentheses make queries easier to read and maintain. -Queries above assume that the `error` [word](#word) is stored in the [log message](./VictoriaLogs/keyConcepts.md#message-field). -If this word is stored in other [field](./VictoriaLogs/keyConcepts.md#data-model) such as `log.level`, then add `log.level:` prefix +Queries above assume that the `error` [word](#word) is stored in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +If this word is stored in other [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) such as `log.level`, then add `log.level:` prefix in front of the `error` word: ```logsql @@ -172,8 +172,8 @@ _time:5m log.level:error !app:(buggy_app OR foobar) ``` The `app` field uniquely identifies the application instance if a single instance runs per each unique `app`. -In this case it is recommended associating the `app` field with [log stream fields](./VictoriaLogs/keyConcepts.md#stream-fields) -during [data ingestion](./VictoriaLogs/data-ingestion/README.md). This usually improves both compression rate +In this case it is recommended associating the `app` field with [log stream fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). This usually improves both compression rate and query performance when querying the needed streams via [`_stream` filter](#stream-filter). If the `app` field is associated with the log stream, then the query above can be rewritten to more performant one: @@ -181,8 +181,8 @@ If the `app` field is associated with the log stream, then the query above can b _time:5m log.level:error _stream:{app!~"buggy_app|foobar"} ``` -This query skips scanning for [log messages](./VictoriaLogs/keyConcepts.md#message-field) from `buggy_app` and `foobar` apps. -It inpsects only `log.level` and [`_stream`](./VictoriaLogs/keyConcepts.md#stream-fields) labels. +This query skips scanning for [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) from `buggy_app` and `foobar` apps. +It inpsects only `log.level` and [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) labels. This significantly reduces disk read IO and CPU time needed for performing the query. LogsQL also provides [functions for statistics calculation](#stats-pipe) over the selected logs. For example, the following query returns the number of logs @@ -194,14 +194,14 @@ _time:5m error | stats count() logs_with_error Finally, it is recommended reading [performance tips](#performance-tips). -Now you are familiar with LogsQL basics. See [LogsQL examples](./VictoriaLogs/logsql-examples.md) and [query syntax](#query-syntax) +Now you are familiar with LogsQL basics. See [LogsQL examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [query syntax](#query-syntax) if you want to continue learning LogsQL. ### Key concepts #### Word -LogsQL splits all the [log fields](./VictoriaLogs/keyConcepts.md#data-model) into words +LogsQL splits all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into words delimited by non-word chars such as whitespace, parens, punctuation chars, etc. For example, the `foo: (bar,"тест")!` string is split into `foo`, `bar` and `тест` words. Words can contain arbitrary [utf-8](https://en.wikipedia.org/wiki/UTF-8) chars. These words are taken into account by full-text search filters such as @@ -216,12 +216,12 @@ For example, the following query selects all the logs for the last 5 minutes by _time:5m ``` -Tip: try [`*` filter](./VictoriaLogs/LogsQL.md#any-value-filter), which selects all the logs stored in VictoriaLogs. -Do not worry - this doesn't crash VictoriaLogs, even if the query selects trillions of logs. See [these docs](./VictoriaLogs/querying/README.md#command-line) +Tip: try [`*` filter](https://docs.victoriametrics.com/victorialogs/logsql/#any-value-filter), which selects all the logs stored in VictoriaLogs. +Do not worry - this doesn't crash VictoriaLogs, even if the query selects trillions of logs. See [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) if you are curious why. Additionally to filters, LogQL query may contain arbitrary mix of optional actions for processing the selected logs. These actions are delimited by `|` and are known as [`pipes`](#pipes). -For example, the following query uses [`stats` pipe](#stats-pipe) for returning the number of [log messages](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query uses [`stats` pipe](#stats-pipe) for returning the number of [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with the `error` [word](#word) for the last 5 minutes: ```logsql @@ -235,8 +235,8 @@ See [the list of supported pipes in LogsQL](#pipes). LogsQL supports various filters for searching for log messages (see below). They can be combined into arbitrary complex queries via [logical filters](#logical-filter). -Filters are applied to [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) by default. -If the filter must be applied to other [log field](./VictoriaLogs/keyConcepts.md#data-model), +Filters are applied to [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) by default. +If the filter must be applied to other [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then its' name followed by the colon must be put in front of the filter. For example, if `error` [word filter](#word-filter) must be applied to the `log.level` field, then use `log.level:error` query. @@ -251,38 +251,38 @@ If doubt, it is recommended quoting field names and filter args. The list of LogsQL filters: -- [Time filter](#time-filter) - matches logs with [`_time` field](./VictoriaLogs/keyConcepts.md#time-field) in the given time range -- [Day range filter](#day-range-filter) - matches logs with [`_time` field](./VictoriaLogs/keyConcepts.md#time-field) in the given per-day time range -- [Week range filter](#week-range-filter) - matches logs with [`_time` field](./VictoriaLogs/keyConcepts.md#time-field) in the given per-week day range -- [Stream filter](#stream-filter) - matches logs, which belong to the given [streams](./VictoriaLogs/keyConcepts.md#stream-fields) +- [Time filter](#time-filter) - matches logs with [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) in the given time range +- [Day range filter](#day-range-filter) - matches logs with [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) in the given per-day time range +- [Week range filter](#week-range-filter) - matches logs with [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) in the given per-week day range +- [Stream filter](#stream-filter) - matches logs, which belong to the given [streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) - [Word filter](#word-filter) - matches logs with the given [word](#word) - [Phrase filter](#phrase-filter) - matches logs with the given phrase - [Prefix filter](#prefix-filter) - matches logs with the given word prefix or phrase prefix - [Substring filter](#substring-filter) - matches logs with the given substring - [Range comparison filter](#range-comparison-filter) - matches logs with field values in the provided range -- [Empty value filter](#empty-value-filter) - matches logs without the given [log field](./VictoriaLogs/keyConcepts.md#data-model) -- [Any value filter](#any-value-filter) - matches logs with the given non-empty [log field](./VictoriaLogs/keyConcepts.md#data-model) +- [Empty value filter](#empty-value-filter) - matches logs without the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) +- [Any value filter](#any-value-filter) - matches logs with the given non-empty [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) - [Exact filter](#exact-filter) - matches logs with the exact value - [Exact prefix filter](#exact-prefix-filter) - matches logs starting with the given prefix - [Multi-exact filter](#multi-exact-filter) - matches logs with one of the specified exact values - [Case-insensitive filter](#case-insensitive-filter) - matches logs with the given case-insensitive word, phrase or prefix - [Sequence filter](#sequence-filter) - matches logs with the given sequence of words or phrases - [Regexp filter](#regexp-filter) - matches logs for the given regexp -- [Range filter](#range-filter) - matches logs with numeric [field values](./VictoriaLogs/keyConcepts.md#data-model) in the given range -- [IPv4 range filter](#ipv4-range-filter) - matches logs with ip address [field values](./VictoriaLogs/keyConcepts.md#data-model) in the given range -- [String range filter](#string-range-filter) - matches logs with [field values](./VictoriaLogs/keyConcepts.md#data-model) in the given string range -- [Length range filter](#length-range-filter) - matches logs with [field values](./VictoriaLogs/keyConcepts.md#data-model) of the given length range +- [Range filter](#range-filter) - matches logs with numeric [field values](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in the given range +- [IPv4 range filter](#ipv4-range-filter) - matches logs with ip address [field values](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in the given range +- [String range filter](#string-range-filter) - matches logs with [field values](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in the given string range +- [Length range filter](#length-range-filter) - matches logs with [field values](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) of the given length range - [Logical filter](#logical-filter) - allows combining other filters ### Time filter -VictoriaLogs scans all the logs per each query if it doesn't contain the filter on [`_time` field](./VictoriaLogs/keyConcepts.md#time-field). +VictoriaLogs scans all the logs per each query if it doesn't contain the filter on [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field). It uses various optimizations in order to accelerate full scan queries without the `_time` filter, but such queries can be slow if the storage contains large number of logs over long time range. The easiest way to optimize queries -is to narrow down the search with the filter on [`_time` field](./VictoriaLogs/keyConcepts.md#time-field). +is to narrow down the search with the filter on [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field). -For example, the following query returns [log messages](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query returns [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) ingested into VictoriaLogs during the last hour, which contain the `error` [word](#word): ```logsql @@ -302,10 +302,10 @@ The following formats are supported for `_time` filter: - `_time:YYYY-MM-DDTHH:MM` - matches all the logs for the particular minute by UTC. For example, `_time:2023-04-25T22:45` matches logs on April 25, 2023 at 22:45 by UTC. - `_time:YYYY-MM-DDTHH:MM:SS` - matches all the logs for the particular second by UTC. For example, `_time:2023-04-25T22:45:59` matches logs on April 25, 2023 at 22:45:59 by UTC. - `_time:[min_time, max_time]` - matches logs on the time range `[min_time, max_time]`, including both `min_time` and `max_time`. - The `min_time` and `max_time` can contain any format specified [here](./README.md#timestamp-formats). + The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/#timestamp-formats). For example, `_time:[2023-04-01, 2023-04-30]` matches logs for the whole April, 2023 by UTC, e.g. it is equivalent to `_time:2023-04`. - `_time:[min_time, max_time)` - matches logs on the time range `[min_time, max_time)`, not including `max_time`. - The `min_time` and `max_time` can contain any format specified [here](./README.md#timestamp-formats). + The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/#timestamp-formats). For example, `_time:[2023-02-01, 2023-03-01)` matches logs for the whole February, 2023 by UTC, e.g. it is equivalent to `_time:2023-02`. It is possible to specify time zone offset for all the absolute time formats by appending `+hh:mm` or `-hh:mm` suffix. @@ -425,11 +425,11 @@ See also: ### Stream filter -VictoriaLogs provides an optimized way to select logs, which belong to particular [log streams](./VictoriaLogs/keyConcepts.md#stream-fields). +VictoriaLogs provides an optimized way to select logs, which belong to particular [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). This can be done via `_stream:{...}` filter. The `{...}` may contain arbitrary -[Prometheus-compatible label selector](./keyConcepts.md#filtering) -over fields associated with [log streams](./VictoriaLogs/keyConcepts.md#stream-fields). -For example, the following query selects [log entries](./VictoriaLogs/keyConcepts.md#data-model) +[Prometheus-compatible label selector](https://docs.victoriametrics.com/keyconcepts/#filtering) +over fields associated with [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). +For example, the following query selects [log entries](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with `app` field equal to `nginx`: ```logsql @@ -460,10 +460,10 @@ See also: ### _stream_id filter -Every [log stream](./VictoriaLogs/keyConcepts.md#stream-fields) in VictoriaMetrics is uniquely identified by `_stream_id` field. +Every [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) in VictoriaMetrics is uniquely identified by `_stream_id` field. The `_stream_id:...` filter allows quickly selecting all the logs belonging to the particular stream. -For example, the following query selects all the logs, which belong to the [log stream](./VictoriaLogs/keyConcepts.md#stream-fields) +For example, the following query selects all the logs, which belong to the [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with `_stream_id` equal to `0000007b000001c850d9950ea6196b1a4812081265faa1c7`: ```logsql @@ -484,8 +484,8 @@ _stream_id:in(0000007b000001c850d9950ea6196b1a4812081265faa1c7, 1230007b456701c8 ``` It is also possible specifying subquery inside `in(...)`, which selects the needed `_stream_id` values. For example, the following query returns -logs for [log streams](./VictoriaLogs/keyConcepts.md#stream-fields) containing `error` [word](#word) -in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) during the last 5 minutes: +logs for [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) containing `error` [word](#word) +in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) during the last 5 minutes: ```logsql _stream_id:in(_time:5m error | fields _stream_id) @@ -499,13 +499,13 @@ See also: ### Word filter The simplest LogsQL query consists of a single [word](#word) to search in log messages. For example, the following query matches -[log messages](./VictoriaLogs/keyConcepts.md#message-field) with `error` [word](#word) inside them: +[log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with `error` [word](#word) inside them: ```logsql error ``` -This query matches the following [log messages](./VictoriaLogs/keyConcepts.md#message-field): +This query matches the following [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): - `error` - `an error happened` @@ -516,8 +516,8 @@ This query doesn't match the following log messages: - `ERROR`, since the filter is case-sensitive by default. Use `i(error)` for this case. See [these docs](#case-insensitive-filter) for details. - `multiple errors occurred`, since the `errors` word doesn't match `error` word. Use `error*` for this case. See [these docs](#prefix-filter) for details. -By default the given [word](#word) is searched in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). -Specify the [field name](./VictoriaLogs/keyConcepts.md#data-model) in front of the word and put a colon after it +By default the given [word](#word) is searched in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +Specify the [field name](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in front of the word and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the `error` [word](#word) in the `log.level` field: ```logsql @@ -549,14 +549,14 @@ See also: Is you need to search for log messages with the specific phrase inside them, then just wrap the phrase in quotes. The phrase can contain any chars, including whitespace, punctuation, parens, etc. They are taken into account during the search. -For example, the following query matches [log messages](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with `ssh: login fail` phrase inside them: ```logsql "ssh: login fail" ``` -This query matches the following [log messages](./VictoriaLogs/keyConcepts.md#message-field): +This query matches the following [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): - `ERROR: ssh: login fail for user "foobar"` - `ssh: login fail!` @@ -579,8 +579,8 @@ logs with `"foo":"bar"` phrase: '"foo":"bar"' ``` -By default the given phrase is searched in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). -Specify the [field name](./VictoriaLogs/keyConcepts.md#data-model) in front of the phrase and put a colon after it +By default the given phrase is searched in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +Specify the [field name](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in front of the phrase and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the `cannot open file` phrase in the `event.original` field: ```logsql @@ -611,13 +611,13 @@ See also: ### Prefix filter If you need to search for log messages with [words](#word) / phrases containing some prefix, then just add `*` char to the end of the [word](#word) / phrase in the query. -For example, the following query returns [log messages](./VictoriaLogs/keyConcepts.md#message-field), which contain [words](#word) with `err` prefix: +For example, the following query returns [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), which contain [words](#word) with `err` prefix: ```logsql err* ``` -This query matches the following [log messages](./VictoriaLogs/keyConcepts.md#message-field): +This query matches the following [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): - `err: foobar` - `cannot open file: error occurred` @@ -628,13 +628,13 @@ This query doesn't match the following log messages: - `fooerror`, since the `fooerror` [word](#word) doesn't start with `err`. Use `~"err"` for this case. See [these docs](#substring-filter) for details. Prefix filter can be applied to [phrases](#phrase-filter). For example, the following query matches -[log messages](./VictoriaLogs/keyConcepts.md#message-field) containing phrases with `unexpected fail` prefix: +[log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) containing phrases with `unexpected fail` prefix: ```logsql "unexpected fail"* ``` -This query matches the following [log messages](./VictoriaLogs/keyConcepts.md#message-field): +This query matches the following [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): - `unexpected fail: IO error` - `error:unexpected failure` @@ -653,8 +653,8 @@ logs with `"foo":"bar` prefix: '"foo":"bar'* ``` -By default the prefix filter is applied to the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). -Specify the needed [field name](./VictoriaLogs/keyConcepts.md#data-model) in front of the prefix filter +By default the prefix filter is applied to the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +Specify the needed [field name](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in front of the prefix filter in order to apply it to the given field. For example, the following query matches `log.level` field containing any word with the `err` prefix: ```logsql @@ -687,7 +687,7 @@ See also: ### Substring filter If it is needed to find logs with some substring, then `~"substring"` filter can be used. For example, the following query matches log entries, -which contain `ampl` text in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field): +which contain `ampl` text in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): ```logsql ~"ampl" @@ -712,7 +712,7 @@ See also: ### Range comparison filter -LogsQL supports `field:>X`, `field:>=X`, `field:X`, `field:>=X`, `field: seconds")` [transformation](#transformations) and then apply the `range()` [filter pipe](#filter-pipe) to the extracted `request_duration` field. Performance tips: -- It is better to query pure numeric [field](./VictoriaLogs/keyConcepts.md#data-model) +- It is better to query pure numeric [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) instead of extracting numeric field from text field via [transformations](#transformations) at query time. - See [other performance tips](#performance-tips). @@ -1113,29 +1113,29 @@ user.ip:ipv4_range("127.0.0.0/8") ``` If you need matching a single IPv4 address, then just put it inside `ipv4_range()`. For example, the following query matches `1.2.3.4` IP -at `user.ip` [field](./VictoriaLogs/keyConcepts.md#data-model): +at `user.ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model): ```logsql user.ip:ipv4_range("1.2.3.4") ``` Note that the `ipv4_range()` doesn't match a string with IPv4 address if this string contains other text. For example, `ipv4_range("127.0.0.0/24")` -doesn't match `request from 127.0.0.1: done` [log message](./VictoriaLogs/keyConcepts.md#message-field), +doesn't match `request from 127.0.0.1: done` [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), since the `127.0.0.1` ip is surrounded by other text. Extract the IP from the message with `parse(_msg, "request from : done")` [transformation](#transformations) and then apply the `ipv4_range()` [filter pipe](#filter-pipe) to the extracted `ip` field. Hints: -- If you need searching for [log messages](./VictoriaLogs/keyConcepts.md#message-field) containing the given `X.Y.Z.Q` IPv4 address, +- If you need searching for [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) containing the given `X.Y.Z.Q` IPv4 address, then `"X.Y.Z.Q"` query can be used. See [these docs](#phrase-filter) for details. -- If you need searching for [log messages](./VictoriaLogs/keyConcepts.md#message-field) containing +- If you need searching for [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) containing at least a single IPv4 address out of the given list, then `"ip1" OR "ip2" ... OR "ipN"` query can be used. See [these docs](#logical-filter) for details. - If you need finding log entries with `ip` field in multiple ranges, then use `ip:(ipv4_range(range1) OR ipv4_range(range2) ... OR ipv4_range(rangeN))` query. See [these docs](#logical-filter) for details. Performance tips: -- It is better querying pure IPv4 [field](./VictoriaLogs/keyConcepts.md#data-model) +- It is better querying pure IPv4 [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) instead of extracting IPv4 from text field via [transformations](#transformations) at query time. - See [other performance tips](#performance-tips). @@ -1171,7 +1171,7 @@ See also: ### Length range filter If you need to filter log message by its length, then `len_range()` filter can be used. -For example, the following LogsQL query matches [log messages](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following LogsQL query matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with lengths in the range `[5, 10]` chars: ```logsql @@ -1188,7 +1188,7 @@ This query doesn't match the following log messages: - `foo`, since it is too short - `foo bar baz abc`, sinc it is too long -It is possible to use `inf` as the upper bound. For example, the following query matches [log messages](./VictoriaLogs/keyConcepts.md#message-field) +It is possible to use `inf` as the upper bound. For example, the following query matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with the length bigger or equal to 5 chars: ```logsql @@ -1201,8 +1201,8 @@ The range boundaries can be expressed in the following forms: - Binary form. Form example, `len_range(0b100110, 0b11111101)` - Integer form with `_` delimiters for better readability. For example, `len_range(1_000, 2_345_678)`. -By default the `len_range()` is applied to the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). -Put the [field name](./VictoriaLogs/keyConcepts.md#data-model) in front of the `len_range()` in order to apply +By default the `len_range()` is applied to the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +Put the [field name](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) in front of the `len_range()` in order to apply the filter to the needed field. For example, the following query matches log entries with the `foo` field length in the range `[10, 20]` chars: ```logsql @@ -1220,34 +1220,34 @@ See also: Simpler LogsQL [filters](#filters) can be combined into more complex filters with the following logical operations: - `q1 AND q2` - matches common log entries returned by both `q1` and `q2`. Arbitrary number of [filters](#filters) can be combined with `AND` operation. - For example, `error AND file AND app` matches [log messages](./VictoriaLogs/keyConcepts.md#message-field), + For example, `error AND file AND app` matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), which simultaneously contain `error`, `file` and `app` [words](#word). The `AND` operation is frequently used in LogsQL queries, so it is allowed to skip the `AND` word. For example, `error file app` is equivalent to `error AND file AND app`. - `q1 OR q2` - merges log entries returned by both `q1` and `q2`. Arbitrary number of [filters](#filters) can be combined with `OR` operation. - For example, `error OR warning OR info` matches [log messages](./VictoriaLogs/keyConcepts.md#message-field), + For example, `error OR warning OR info` matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), which contain at least one of `error`, `warning` or `info` [words](#word). - `NOT q` - returns all the log entries except of those which match `q`. For example, `NOT info` returns all the - [log messages](./VictoriaLogs/keyConcepts.md#message-field), + [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), which do not contain `info` [word](#word). The `NOT` operation is frequently used in LogsQL queries, so it is allowed substituting `NOT` with `!` in queries. For example, `!info` is equivalent to `NOT info`. The `NOT` operation has the highest priority, `AND` has the middle priority and `OR` has the lowest priority. The priority order can be changed with parentheses. For example, `NOT info OR debug` is interpreted as `(NOT info) OR debug`, -so it matches [log messages](./VictoriaLogs/keyConcepts.md#message-field), +so it matches [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), which do not contain `info` [word](#word), while it also matches messages with `debug` word (which may contain the `info` word). This is not what most users expect. In this case the query can be rewritten to `NOT (info OR debug)`, which correctly returns log messages without `info` and `debug` [words](#word). LogsQL supports arbitrary complex logical queries with arbitrary mix of `AND`, `OR` and `NOT` operations and parentheses. -By default logical filters apply to the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) -unless the inner filters explicitly specify the needed [log field](./VictoriaLogs/keyConcepts.md#data-model) via `field_name:filter` syntax. +By default logical filters apply to the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) +unless the inner filters explicitly specify the needed [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) via `field_name:filter` syntax. For example, `(error OR warn) AND host.hostname:host123` is interpreted as `(_msg:error OR _msg:warn) AND host.hostname:host123`. -It is possible to specify a single [log field](./VictoriaLogs/keyConcepts.md#data-model) for multiple filters +It is possible to specify a single [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) for multiple filters with the following syntax: ```logsql @@ -1261,7 +1261,7 @@ Performance tips: - VictoriaLogs executes logical operations from the left to the right, so it is recommended moving the most specific and the fastest filters (such as [word filter](#word-filter) and [phrase filter](#phrase-filter)) to the left, while moving less specific and the slowest filters (such as [regexp filter](#regexp-filter) and [case-insensitive filter](#case-insensitive-filter)) - to the right. For example, if you need to find [log messages](./VictoriaLogs/keyConcepts.md#message-field) + to the right. For example, if you need to find [log messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with the `error` word, which match some `/foo/(bar|baz)` regexp, it is better from performance PoV to use the query `error ~"/foo/(bar|baz)"` instead of `~"/foo/(bar|baz)" error`. @@ -1273,7 +1273,7 @@ Performance tips: Additionally to [filters](#filters), LogsQL query may contain arbitrary mix of '|'-delimited actions known as `pipes`. For example, the following query uses [`stats`](#stats-pipe), [`sort`](#sort-pipe) and [`limit`](#limit-pipe) pipes -for returning top 10 [log streams](./VictoriaLogs/keyConcepts.md#stream-fields) +for returning top 10 [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with the biggest number of logs during the last 5 minutes: ```logsql @@ -1282,38 +1282,38 @@ _time:5m | stats by (_stream) count() per_stream_logs | sort by (per_stream_logs LogsQL supports the following pipes: -- [`copy`](#copy-pipe) copies [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`delete`](#delete-pipe) deletes [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`drop_empty_fields`](#drop_empty_fields-pipe) drops [log fields](./VictoriaLogs/keyConcepts.md#data-model) with empty values. +- [`copy`](#copy-pipe) copies [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`delete`](#delete-pipe) deletes [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`drop_empty_fields`](#drop_empty_fields-pipe) drops [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with empty values. - [`extract`](#extract-pipe) extracts the specified text into the given log fields. - [`extract_regexp`](#extract_regexp-pipe) extracts the specified text into the given log fields via [RE2 regular expressions](https://github.com/google/re2/wiki/Syntax). -- [`field_names`](#field_names-pipe) returns all the names of [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`field_values`](#field_values-pipe) returns all the values for the given [log field](./VictoriaLogs/keyConcepts.md#data-model). -- [`fields`](#fields-pipe) selects the given set of [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`field_names`](#field_names-pipe) returns all the names of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`field_values`](#field_values-pipe) returns all the values for the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`fields`](#fields-pipe) selects the given set of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - [`filter`](#filter-pipe) applies additional [filters](#filters) to results. -- [`format`](#format-pipe) formats output field from input [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`format`](#format-pipe) formats output field from input [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - [`limit`](#limit-pipe) limits the number selected logs. -- [`math`](#math-pipe) performs mathematical calculations over [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`math`](#math-pipe) performs mathematical calculations over [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - [`offset`](#offset-pipe) skips the given number of selected logs. -- [`pack_json`](#pack_json-pipe) packs [log fields](./VictoriaLogs/keyConcepts.md#data-model) into JSON object. -- [`pack_logfmt`](#pack_logfmt-pipe) packs [log fields](./VictoriaLogs/keyConcepts.md#data-model) into [logfmt](https://brandur.org/logfmt) message. -- [`rename`](#rename-pipe) renames [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`replace`](#replace-pipe) replaces substrings in the specified [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`replace_regexp`](#replace_regexp-pipe) updates [log fields](./VictoriaLogs/keyConcepts.md#data-model) with regular expressions. -- [`sort`](#sort-pipe) sorts logs by the given [fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`pack_json`](#pack_json-pipe) packs [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into JSON object. +- [`pack_logfmt`](#pack_logfmt-pipe) packs [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into [logfmt](https://brandur.org/logfmt) message. +- [`rename`](#rename-pipe) renames [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`replace`](#replace-pipe) replaces substrings in the specified [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`replace_regexp`](#replace_regexp-pipe) updates [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with regular expressions. +- [`sort`](#sort-pipe) sorts logs by the given [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - [`stats`](#stats-pipe) calculates various stats over the selected logs. - [`stream_context`](#stream_context-pipe) allows selecting surrounding logs in front and after the matching logs - per each [log stream](./VictoriaLogs/keyConcepts.md#stream-fields). + per each [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). - [`top`](#top-pipe) returns top `N` field sets with the maximum number of matching logs. - [`uniq`](#uniq-pipe) returns unique log entires. -- [`unpack_json`](#unpack_json-pipe) unpacks JSON messages from [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`unpack_logfmt`](#unpack_logfmt-pipe) unpacks [logfmt](https://brandur.org/logfmt) messages from [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`unpack_syslog`](#unpack_syslog-pipe) unpacks [syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`unroll`](#unroll-pipe) unrolls JSON arrays from [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`unpack_json`](#unpack_json-pipe) unpacks JSON messages from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`unpack_logfmt`](#unpack_logfmt-pipe) unpacks [logfmt](https://brandur.org/logfmt) messages from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`unpack_syslog`](#unpack_syslog-pipe) unpacks [syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`unroll`](#unroll-pipe) unrolls JSON arrays from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). ### copy pipe -If some [log fields](./VictoriaLogs/keyConcepts.md#data-model) must be copied, then `| copy src1 as dst1, ..., srcN as dstN` [pipe](#pipes) can be used. +If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) must be copied, then `| copy src1 as dst1, ..., srcN as dstN` [pipe](#pipes) can be used. For example, the following query copies `host` field to `server` for logs over the last 5 minutes, so the output contains both `host` and `server` fields: ```logsql @@ -1321,7 +1321,7 @@ _time:5m | copy host as server ``` Multiple fields can be copied with a single `| copy ...` pipe. For example, the following query copies -[`_time` field](./VictoriaLogs/keyConcepts.md#time-field) to `timestamp`, while [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +[`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) to `timestamp`, while [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) is copied to `message`: ```logsql @@ -1340,7 +1340,7 @@ See also: ### delete pipe -If some [log fields](./VictoriaLogs/keyConcepts.md#data-model) must be deleted, then `| delete field1, ..., fieldN` [pipe](#pipes) can be used. +If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) must be deleted, then `| delete field1, ..., fieldN` [pipe](#pipes) can be used. For example, the following query deletes `host` and `app` fields from the logs over the last 5 minutes: ```logsql @@ -1356,7 +1356,7 @@ See also: ### drop_empty_fields pipe -`| drop_empty_fields` pipe drops [fields](./VictoriaLogs/keyConcepts.md#data-model) with empty values. It also skips log entries with zero non-empty fields. +`| drop_empty_fields` pipe drops [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with empty values. It also skips log entries with zero non-empty fields. For example, the following query drops possible empty `email` field generated by [`extract` pipe](#extract-pipe) if the `foo` field doesn't contain email: @@ -1373,12 +1373,12 @@ See also: ### extract pipe `| extract "pattern" from field_name` [pipe](#pipes) allows extracting arbitrary text into output fields according to the [`pattern`](#format-for-extract-pipe-pattern) from the given -[`field_name`](./VictoriaLogs/keyConcepts.md#data-model). Existing log fields remain unchanged after the `| extract ...` pipe. +[`field_name`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). Existing log fields remain unchanged after the `| extract ...` pipe. `| extract ...` can be useful for extracting additional fields needed for further data processing with other pipes such as [`stats` pipe](#stats-pipe) or [`sort` pipe](#sort-pipe). For example, the following query selects logs with the `error` [word](#word) for the last day, -extracts ip address from [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) into `ip` field and then calculates top 10 ip addresses +extracts ip address from [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) into `ip` field and then calculates top 10 ip addresses with the biggest number of logs: ```logsql @@ -1388,7 +1388,7 @@ _time:1d error | extract "ip= " from _msg | stats by (ip) count() logs | sor It is expected that `_msg` field contains `ip=...` substring ending with space. For example, `error ip=1.2.3.4 from user_id=42`. If there is no such substring in the current `_msg` field, then the `ip` output field will be empty. -If the `| extract ...` pipe is applied to [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field), then the `from _msg` part can be omitted. +If the `| extract ...` pipe is applied to [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), then the `from _msg` part can be omitted. For example, the following query is equivalent to the previous one: ```logsql @@ -1452,7 +1452,7 @@ The empty string values can be dropped with [`drop_empty_fields` pipe](#drop_emp Matching finishes successfully when `textN+1` is found in the input text. If the `pattern` ends with `` and doesn't contain `textN+1`, then the `` matches the remaining input text. -For example, if [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) contains the following text: +For example, if [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) contains the following text: ``` 1.2.3.4 GET /foo/bar?baz 404 "Mozilla foo bar baz" some tail here @@ -1498,8 +1498,8 @@ For example, the following `pattern` properly matches `a < b` text by extracting If some log entries must be skipped from [`extract` pipe](#extract-pipe), then add `if ()` filter after the `extract` word. The `` can contain arbitrary [filters](#filters). For example, the following query extracts `ip` field -from [`_msg` field](./VictoriaLogs/keyConcepts.md#data-model) only -if the input [log entry](./VictoriaLogs/keyConcepts.md#data-model) doesn't contain `ip` field or this field is empty: +from [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) only +if the input [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) doesn't contain `ip` field or this field is empty: ```logsql _time:5m | extract if (ip:"") "ip= " @@ -1514,18 +1514,18 @@ _time:5m | extract "ip= " keep_original_fields ### extract_regexp pipe -`| extract_regexp "pattern" from field_name` [pipe](#pipes) extracts substrings from the [`field_name` field](./VictoriaLogs/keyConcepts.md#data-model) +`| extract_regexp "pattern" from field_name` [pipe](#pipes) extracts substrings from the [`field_name` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) according to the provided `pattern`, and stores them into field names according to the named fields inside the `pattern`. The `pattern` must contain [RE2 regular expression](https://github.com/google/re2/wiki/Syntax) with named fields (aka capturing groups) in the form `(?P...)`. -Matching substrings are stored to the given `capture_field_name` [log fields](./VictoriaLogs/keyConcepts.md#data-model). -For example, the following query extracts ipv4 addresses from [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +Matching substrings are stored to the given `capture_field_name` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +For example, the following query extracts ipv4 addresses from [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) and puts them into `ip` field for logs over the last 5 minutes: ```logsql _time:5m | extract_regexp "(?P([0-9]+[.]){3}[0-9]+)" from _msg ``` -The `from _msg` part can be omitted if the data extraction is performed from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `from _msg` part can be omitted if the data extraction is performed from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). So the following query is equivalent to the previous one: ```logsql @@ -1560,8 +1560,8 @@ See also: If some log entries must be skipped from [`extract_regexp` pipe](#extract-pipe), then add `if ()` filter after the `extract` word. The `` can contain arbitrary [filters](#filters). For example, the following query extracts `ip` -from [`_msg` field](./VictoriaLogs/keyConcepts.md#data-model) only -if the input [log entry](./VictoriaLogs/keyConcepts.md#data-model) doesn't contain `ip` field or this field is empty: +from [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) only +if the input [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) doesn't contain `ip` field or this field is empty: ```logsql _time:5m | extract_regexp if (ip:"") "ip=(?P([0-9]+[.]){3}[0-9]+)" @@ -1576,7 +1576,7 @@ _time:5m | extract_regexp "ip=(?P([0-9]+[.]){3}[0-9]+)" keep_original_fields ### field_names pipe -`| field_names` [pipe](#pipes) returns all the names of [log fields](./VictoriaLogs/keyConcepts.md#data-model) +`| field_names` [pipe](#pipes) returns all the names of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with an estimated number of logs per each field name. For example, the following query returns all the field names with the number of matching logs over the last 5 minutes: @@ -1593,7 +1593,7 @@ See also: ### field_values pipe -`| field_values field_name` [pipe](#pipe) returns all the values for the given [`field_name` field](./VictoriaLogs/keyConcepts.md#data-model) +`| field_values field_name` [pipe](#pipe) returns all the values for the given [`field_name` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the number of logs per each value. For example, the following query returns all the values with the number of matching logs for the field `level` over logs for the last 5 minutes: @@ -1618,9 +1618,9 @@ See also: ### fields pipe -By default all the [log fields](./VictoriaLogs/keyConcepts.md#data-model) are returned in the response. +By default all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) are returned in the response. It is possible to select the given set of log fields with `| fields field1, ..., fieldN` [pipe](#pipes). For example, the following query selects only `host` -and [`_msg`](./VictoriaLogs/keyConcepts.md#message-field) fields from logs for the last 5 minutes: +and [`_msg`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) fields from logs for the last 5 minutes: ```logsql _time:5m | fields host, _msg @@ -1642,7 +1642,7 @@ See also: The `| filter ...` [pipe](#pipes) allows filtering the selected logs entries with arbitrary [filters](#filters). -For example, the following query returns `host` [field](./VictoriaLogs/keyConcepts.md#data-model) values +For example, the following query returns `host` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) values if the number of log messages with the `error` [word](#word) for them over the last hour exceeds `1_000`: ```logsql @@ -1669,17 +1669,17 @@ See also: ### format pipe -`| format "pattern" as result_field` [pipe](#pipe) combines [log fields](./VictoriaLogs/keyConcepts.md#data-model) +`| format "pattern" as result_field` [pipe](#pipe) combines [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) according to the `pattern` and stores it to the `result_field`. -For example, the following query stores `request from :` text into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field), -by substituting `` and `` with the corresponding [log field](./VictoriaLogs/keyConcepts.md#data-model) values: +For example, the following query stores `request from :` text into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), +by substituting `` and `` with the corresponding [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) values: ```logsql _time:5m | format "request from :" as _msg ``` -If the result of the `format` pattern is stored into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field), +If the result of the `format` pattern is stored into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), then `as _msg` part can be omitted. The following query is equivalent to the previous one: ```logsql @@ -1687,7 +1687,7 @@ _time:5m | format "request from :" ``` If some field values must be put into double quotes before formatting, then add `q:` in front of the corresponding field name. -For example, the following command generates properly encoded JSON object from `_msg` and `stacktrace` [log fields](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following command generates properly encoded JSON object from `_msg` and `stacktrace` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) and stores it into `my_json` output field: ```logsql @@ -1733,10 +1733,10 @@ See also: #### Conditional format -If the [`format` pipe](#format-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`format` pipe](#format-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` just after the `format` word. The `` can contain arbitrary [filters](#filters). For example, the following query stores the formatted result to `message` field -only if `ip` and `host` [fields](./VictoriaLogs/keyConcepts.md#data-model) aren't empty: +only if `ip` and `host` [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) aren't empty: ```logsql _time:5m | format if (ip:* and host:*) "request from :" as message @@ -1769,7 +1769,7 @@ See also: ### math pipe -`| math ...` [pipe](#pipes) performs mathematical calculations over [numeric values](#numeric-values) stored in [log fields](./VictoriaLogs/keyConcepts.md#data-model). +`| math ...` [pipe](#pipes) performs mathematical calculations over [numeric values](#numeric-values) stored in [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). It has the following format: ``` @@ -1815,7 +1815,7 @@ The following mathematical operations are supported by `math` pipe: Every `argX` argument in every mathematical operation can contain one of the following values: -- The name of [log field](./VictoriaLogs/keyConcepts.md#data-model). For example, `errors_total / requests_total`. +- The name of [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, `errors_total / requests_total`. The log field is parsed into numeric value if it contains [supported numeric value](#numeric-values). The log field is parsed into [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds if it contains [rfc3339 time](https://www.rfc-editor.org/rfc/rfc3339). The log field is parsed into `uint32` number if it contains IPv4 address. The log field is parsed into `NaN` in other cases. @@ -1823,14 +1823,14 @@ Every `argX` argument in every mathematical operation can contain one of the fol - Another mathematical expression, which can be put inside `(...)`. For example, `(a + b) * c`. The parsed time, duration and IPv4 address can be converted back to string representation after math transformations with the help of [`format` pipe](#format-pipe). For example, -the following query rounds the `request_duration` [field](./VictoriaLogs/keyConcepts.md#data-model) to seconds before converting it back to string representation: +the following query rounds the `request_duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) to seconds before converting it back to string representation: ```logsql _time:5m | math round(request_duration, 1e9) as request_duration_nsecs | format '' as request_duration ``` The `eval` keyword can be used instead of `math` for convenience. For example, the following query calculates `duration_msecs` field -by multiplying `duration_secs` [field](./VictoriaLogs/keyConcepts.md#data-model) to `1000`: +by multiplying `duration_secs` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) to `1000`: ```logsql _time:5m | eval (duration_secs * 1000) as duration_msecs @@ -1846,7 +1846,7 @@ See also: ### offset pipe If some selected logs must be skipped after [`sort`](#sort-pipe), then `| offset N` [pipe](#pipes) can be used, where `N` can contain any [supported integer numeric value](#numeric-values). -For example, the following query skips the first 100 logs over the last 5 minutes after sorting them by [`_time`](./VictoriaLogs/keyConcepts.md#time-field): +For example, the following query skips the first 100 logs over the last 5 minutes after sorting them by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): ```logsql _time:5m | sort by (_time) | offset 100 @@ -1864,17 +1864,17 @@ See also: ### pack_json pipe -`| pack_json as field_name` [pipe](#pipe) packs all [log fields](./VictoriaLogs/keyConcepts.md#data-model) into JSON object +`| pack_json as field_name` [pipe](#pipe) packs all [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into JSON object and stores it as a string in the given `field_name`. -For example, the following query packs all the fields into JSON object and stores it into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query packs all the fields into JSON object and stores it into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) for logs over the last 5 minutes: ```logsql _time:5m | pack_json as _msg ``` -The `as _msg` part can be omitted if packed JSON object is stored into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `as _msg` part can be omitted if packed JSON object is stored into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -1903,17 +1903,17 @@ See also: ### pack_logfmt pipe -`| pack_logfmt as field_name` [pipe](#pipe) packs all [log fields](./VictoriaLogs/keyConcepts.md#data-model) into [logfmt](https://brandur.org/logfmt) message +`| pack_logfmt as field_name` [pipe](#pipe) packs all [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into [logfmt](https://brandur.org/logfmt) message and stores it as a string in the given `field_name`. For example, the following query packs all the fields into [logfmt](https://brandur.org/logfmt) message and stores it -into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) for logs over the last 5 minutes: +into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) for logs over the last 5 minutes: ```logsql _time:5m | pack_logfmt as _msg ``` -The `as _msg` part can be omitted if packed message is stored into [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `as _msg` part can be omitted if packed message is stored into [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -1941,7 +1941,7 @@ See also: ### rename pipe -If some [log fields](./VictoriaLogs/keyConcepts.md#data-model) must be renamed, then `| rename src1 as dst1, ..., srcN as dstN` [pipe](#pipes) can be used. +If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) must be renamed, then `| rename src1 as dst1, ..., srcN as dstN` [pipe](#pipes) can be used. For example, the following query renames `host` field to `server` for logs over the last 5 minutes, so the output contains `server` field instead of `host` field: ```logsql @@ -1967,16 +1967,16 @@ See also: ### replace pipe `| replace ("old", "new") at field` [pipe](#pipes) replaces all the occurrences of the `old` substring with the `new` substring -in the given [`field`](./VictoriaLogs/keyConcepts.md#data-model). +in the given [`field`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query replaces all the `secret-password` substrings with `***` in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query replaces all the `secret-password` substrings with `***` in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) for logs over the last 5 minutes: ```logsql _time:5m | replace ("secret-password", "***") at _msg ``` -The `at _msg` part can be omitted if the replacement occurs in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `at _msg` part can be omitted if the replacement occurs in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -1984,7 +1984,7 @@ _time:5m | replace ("secret-password", "***") ``` The number of replacements can be limited with `limit N` at the end of `replace`. For example, the following query replaces only the first `foo` substring with `bar` -at the [log field](./VictoriaLogs/keyConcepts.md#data-model) `baz`: +at the [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) `baz`: ```logsql _time:5m | replace ('foo', 'bar') at baz limit 1 @@ -2002,7 +2002,7 @@ See also: #### Conditional replace -If the [`replace` pipe](#replace-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`replace` pipe](#replace-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `replace`. The `` can contain arbitrary [filters](#filters). For example, the following query replaces `secret` with `***` in the `password` field only if `user_type` field equals to `admin`: @@ -2014,18 +2014,18 @@ _time:5m | replace if (user_type:=admin) replace ("secret", "***") at password ### replace_regexp pipe `| replace_regexp ("regexp", "replacement") at field` [pipe](#pipes) replaces all the substrings matching the given `regexp` with the given `replacement` -in the given [`field`](./VictoriaLogs/keyConcepts.md#data-model). +in the given [`field`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). The `regexp` must contain regular expression with [RE2 syntax](https://github.com/google/re2/wiki/Syntax). The `replacement` may contain `$N` or `${N}` placeholders, which are substituted with the `N-th` capturing group in the `regexp`. -For example, the following query replaces all the substrings starting with `host-` and ending with `-foo` with the contents between `host-` and `-foo` in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) for logs over the last 5 minutes: +For example, the following query replaces all the substrings starting with `host-` and ending with `-foo` with the contents between `host-` and `-foo` in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) for logs over the last 5 minutes: ```logsql _time:5m | replace_regexp ("host-(.+?)-foo", "$1") at _msg ``` -The `at _msg` part can be omitted if the replacement occurs in the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `at _msg` part can be omitted if the replacement occurs in the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -2033,7 +2033,7 @@ _time:5m | replace_regexp ("host-(.+?)-foo", "$1") ``` The number of replacements can be limited with `limit N` at the end of `replace`. For example, the following query replaces only the first `password: ...` substring -ending with whitespace with empty substring at the [log field](./VictoriaLogs/keyConcepts.md#data-model) `baz`: +ending with whitespace with empty substring at the [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) `baz`: ```logsql _time:5m | replace_regexp ('password: [^ ]+', '') at baz limit 1 @@ -2054,7 +2054,7 @@ See also: #### Conditional replace_regexp -If the [`replace_regexp` pipe](#replace-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`replace_regexp` pipe](#replace-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `replace_regexp`. The `` can contain arbitrary [filters](#filters). For example, the following query replaces `password: ...` substrings ending with whitespace with `***` in the `foo` field only if `user_type` field equals to `admin`: @@ -2066,11 +2066,11 @@ _time:5m | replace_regexp if (user_type:=admin) replace ("password: [^ ]+", "") ### sort pipe By default logs are selected in arbitrary order because of performance reasons. If logs must be sorted, then `| sort by (field1, ..., fieldN)` [pipe](#pipes) can be used. -The returned logs are sorted by the given [fields](./VictoriaLogs/keyConcepts.md#data-model) +The returned logs are sorted by the given [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) using [natural sorting](https://en.wikipedia.org/wiki/Natural_sort_order). -For example, the following query returns logs for the last 5 minutes sorted by [`_stream`](./VictoriaLogs/keyConcepts.md#stream-fields) -and then by [`_time`](./VictoriaLogs/keyConcepts.md#time-field): +For example, the following query returns logs for the last 5 minutes sorted by [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) +and then by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): ```logsql _time:5m | sort by (_stream, _time) @@ -2098,14 +2098,14 @@ Sorting of big number of logs can consume a lot of CPU time and memory. Sometime or the smallest values. This can be done by adding `limit N` to the end of `sort ...` pipe. Such a query consumes lower amounts of memory when sorting big number of logs, since it keeps in memory only `N` log entries. For example, the following query returns top 10 log entries with the biggest values -for the `request_duration` [field](./VictoriaLogs/keyConcepts.md#data-model) during the last hour: +for the `request_duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) during the last hour: ```logsql _time:1h | sort by (request_duration desc) limit 10 ``` If the first `N` sorted results must be skipped, then `offset N` can be added to `sort` pipe. For example, -the following query skips the first 10 logs with the biggest `request_duration` [field](./VictoriaLogs/keyConcepts.md#data-model), +the following query skips the first 10 logs with the biggest `request_duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), and then returns the next 20 sorted logs for the last 5 minutes: ```logsql @@ -2113,8 +2113,8 @@ _time:1h | sort by (request_duration desc) offset 10 limit 20 ``` It is possible returning a rank (sort order number) for every sorted log by adding `rank as ` to the end of `| sort ...` pipe. -For example, the following query stores rank for sorted by [`_time`](./VictoriaLogs/keyConcepts.md#time-field) logs -into `position` [field](./VictoriaLogs/keyConcepts.md#data-model): +For example, the following query stores rank for sorted by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) logs +into `position` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model): ```logsql _time:5m | sort by (_time) rank as position @@ -2126,7 +2126,7 @@ It is recommended limiting the number of logs before sorting with the following - Adding `limit N` to the end of `sort ...` pipe. - Reducing the selected time range with [time filter](#time-filter). - Using more specific [filters](#filters), so they select less logs. -- Limiting the number of selected [fields](./VictoriaLogs/keyConcepts.md#data-model) via [`fields` pipe](#fields-pipe). +- Limiting the number of selected [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) via [`fields` pipe](#fields-pipe). See also: @@ -2158,7 +2158,7 @@ to store the result of the corresponding stats function. The `as` keyword is opt For example, the following query calculates the following stats for logs over the last 5 minutes: - the number of logs with the help of [`count` stats function](#count-stats); -- the number of unique [log streams](./VictoriaLogs/keyConcepts.md#stream-fields) with the help of [`count_uniq` stats function](#count_uniq-stats): +- the number of unique [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with the help of [`count_uniq` stats function](#count_uniq-stats): ```logsql _time:5m | stats count() logs_total, count_uniq(_stream) streams_total @@ -2203,7 +2203,7 @@ The following LogsQL syntax can be used for calculating independent stats per gr stats_funcN(...) as result_nameN ``` -This calculates `stats_func*` per each `(field1, ..., fieldM)` group of [log fields](./VictoriaLogs/keyConcepts.md#data-model). +This calculates `stats_func*` per each `(field1, ..., fieldM)` group of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, the following query calculates the number of logs and unique ip addresses over the last 5 minutes, grouped by `(host, path)` fields: @@ -2235,7 +2235,7 @@ The following syntax can be used for calculating stats grouped by time buckets: stats_funcN(...) as result_nameN ``` -This calculates `stats_func*` per each `step` of [`_time`](./VictoriaLogs/keyConcepts.md#time-field) field. +This calculates `stats_func*` per each `step` of [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) field. The `step` can have any [duration value](#duration-values). For example, the following LogsQL query returns per-minute number of logs and unique ip addresses over the last 5 minutes: @@ -2258,7 +2258,7 @@ Additionally, the following `step` values are supported: #### Stats by time buckets with timezone offset -VictoriaLogs stores [`_time`](./VictoriaLogs/keyConcepts.md#time-field) values as [Unix time](https://en.wikipedia.org/wiki/Unix_time) +VictoriaLogs stores [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) values as [Unix time](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. This time corresponds to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) time zone. Sometimes it is needed calculating stats grouped by days or weeks at non-UTC timezone. This is possible with the following syntax: @@ -2276,7 +2276,7 @@ _time:1w | stats by (_time:1d offset 2h) count() logs_total Every log field inside `| stats by (...)` can be bucketed in the same way at `_time` field in [this example](#stats-by-time-buckets). Any [numeric value](#numeric-values) can be used as `step` value for the bucket. For example, the following query calculates -the number of requests for the last hour, bucketed by 10KB of `request_size_bytes` [field](./VictoriaLogs/keyConcepts.md#data-model): +the number of requests for the last hour, bucketed by 10KB of `request_size_bytes` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model): ```logsql _time:1h | stats by (request_size_bytes:10KB) count() requests @@ -2284,9 +2284,9 @@ _time:1h | stats by (request_size_bytes:10KB) count() requests #### Stats by IPv4 buckets -Stats can be bucketed by [log field](./VictoriaLogs/keyConcepts.md#data-model) containing [IPv4 addresses](https://en.wikipedia.org/wiki/IP_address) +Stats can be bucketed by [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) containing [IPv4 addresses](https://en.wikipedia.org/wiki/IP_address) via the `ip_field_name:/network_mask` syntax inside `by(...)` clause. For example, the following query returns the number of log entries per `/24` subnetwork -extracted from the `ip` [log field](./VictoriaLogs/keyConcepts.md#data-model) during the last 5 minutes: +extracted from the `ip` [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) during the last 5 minutes: ```logsql _time:5m | stats by (ip:/24) count() requests_per_subnet @@ -2296,7 +2296,7 @@ _time:5m | stats by (ip:/24) count() requests_per_subnet Sometimes it is needed to calculate stats on different subsets of matching logs. This can be done by inserting `if ()` condition between [stats function](#stats-pipe-functions) and `result_name`, where `any_filter` can contain arbitrary [filters](#filters). -For example, the following query calculates individually the number of [logs messages](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query calculates individually the number of [logs messages](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) with `GET`, `POST` and `PUT` [words](#word), additionally to the total number of logs over the last 5 minutes: ```logsql @@ -2309,8 +2309,8 @@ _time:5m | stats ### stream_context pipe -`| stream_context ...` [pipe](#pipes) allows selecting surrounding logs for the matching logs in [logs stream](./VictoriaLogs/keyConcepts.md#stream-fields) -in the way similar to `grep -A` / `grep -B`. The returned log chunks are delimited with `---` [log message](./VictoriaLogs/keyConcepts.md#message-field) +`| stream_context ...` [pipe](#pipes) allows selecting surrounding logs for the matching logs in [logs stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) +in the way similar to `grep -A` / `grep -B`. The returned log chunks are delimited with `---` [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) for easier investigation. For example, the following query returns up to 10 additional logs after every log message with the `panic` [word](#word) across all the logs for the last 5 minutes: @@ -2336,10 +2336,10 @@ The `| stream_context` [pipe](#pipes) must go first just after the [filters](#fi ### top pipe -`| top N by (field1, ..., fieldN)` [pipe](#pipes) returns top `N` sets for `(field1, ..., fieldN)` [log fields](./VictoriaLogs/keyConcepts.md#data-model) +`| top N by (field1, ..., fieldN)` [pipe](#pipes) returns top `N` sets for `(field1, ..., fieldN)` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the maximum number of matching log entries. -For example, the following query returns top 7 [log streams](./VictoriaLogs/keyConcepts.md#stream-fields) +For example, the following query returns top 7 [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with the maximum number of log entries over the last 5 minutes: ```logsql @@ -2347,7 +2347,7 @@ _time:5m | top 7 by (_stream) ``` The `N` is optional. If it is skipped, then top 10 entries are returned. For example, the following query returns top 10 values -for `ip` [field](./VictoriaLogs/keyConcepts.md#data-model) seen in logs for the last 5 minutes: +for `ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) seen in logs for the last 5 minutes: ```logsql _time:5m | top by (ip) @@ -2369,7 +2369,7 @@ See also: ### uniq pipe `| uniq ...` [pipe](#pipes) returns unique results over the selected logs. For example, the following LogsQL query -returns unique values for `ip` [log field](./VictoriaLogs/keyConcepts.md#data-model) +returns unique values for `ip` [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2415,18 +2415,18 @@ See also: ### unpack_json pipe -`| unpack_json from field_name` [pipe](#pipes) unpacks `{"k1":"v1", ..., "kN":"vN"}` JSON from the given input [`field_name`](./VictoriaLogs/keyConcepts.md#data-model) +`| unpack_json from field_name` [pipe](#pipes) unpacks `{"k1":"v1", ..., "kN":"vN"}` JSON from the given input [`field_name`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into `k1`, ... `kN` output field names with the corresponding `v1`, ..., `vN` values. It overrides existing fields with names from the `k1`, ..., `kN` list. Other fields remain untouched. -Nested JSON is unpacked according to the rules defined [here](./VictoriaLogs/keyConcepts.md#data-model). +Nested JSON is unpacked according to the rules defined [here](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query unpacks JSON fields from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) across logs for the last 5 minutes: +For example, the following query unpacks JSON fields from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) across logs for the last 5 minutes: ```logsql _time:5m | unpack_json from _msg ``` -The `from _msg` part can be omitted when JSON fields are unpacked from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `from _msg` part can be omitted when JSON fields are unpacked from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -2434,7 +2434,7 @@ _time:5m | unpack_json ``` If only some fields must be extracted from JSON, then they can be enumerated inside `fields (...)`. For example, the following query unpacks only `foo` and `bar` -fields from JSON value stored in `my_json` [log field](./VictoriaLogs/keyConcepts.md#data-model): +fields from JSON value stored in `my_json` [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model): ```logsql _time:5m | unpack_json from my_json fields (foo, bar) @@ -2455,7 +2455,7 @@ _time:5m | unpack_json fields (ip, host) skip_empty_results ``` Performance tip: if you need extracting a single field from long JSON, it is faster to use [`extract` pipe](#extract-pipe). For example, the following query extracts `"ip"` field from JSON -stored in [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) at the maximum speed: +stored in [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) at the maximum speed: ``` _time:5m | extract '"ip":' @@ -2472,7 +2472,7 @@ _time:5m | unpack_json from foo result_prefix "foo_" Performance tips: - It is better from performance and resource usage PoV ingesting parsed JSON logs into VictoriaLogs - according to the [supported data model](./VictoriaLogs/keyConcepts.md#data-model) + according to the [supported data model](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) instead of ingesting unparsed JSON lines into VictoriaLogs and then parsing them at query time with [`unpack_json` pipe](#unpack_json-pipe). - It is recommended using more specific [log filters](#filters) in order to reduce the number of log entries, which are passed to `unpack_json`. @@ -2490,7 +2490,7 @@ See also: #### Conditional unpack_json -If the [`unpack_json` pipe](#unpack_json-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`unpack_json` pipe](#unpack_json-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `unpack_json`. The `` can contain arbitrary [filters](#filters). For example, the following query unpacks JSON fields from `foo` field only if `ip` field in the current log entry isn't set or empty: @@ -2501,17 +2501,17 @@ _time:5m | unpack_json if (ip:"") from foo ### unpack_logfmt pipe `| unpack_logfmt from field_name` [pipe](#pipes) unpacks `k1=v1 ... kN=vN` [logfmt](https://brandur.org/logfmt) fields -from the given [`field_name`](./VictoriaLogs/keyConcepts.md#data-model) into `k1`, ... `kN` field names +from the given [`field_name`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into `k1`, ... `kN` field names with the corresponding `v1`, ..., `vN` values. It overrides existing fields with names from the `k1`, ..., `kN` list. Other fields remain untouched. -For example, the following query unpacks [logfmt](https://brandur.org/logfmt) fields from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query unpacks [logfmt](https://brandur.org/logfmt) fields from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) across logs for the last 5 minutes: ```logsql _time:5m | unpack_logfmt from _msg ``` -The `from _msg` part can be omitted when [logfmt](https://brandur.org/logfmt) fields are unpacked from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +The `from _msg` part can be omitted when [logfmt](https://brandur.org/logfmt) fields are unpacked from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -2541,7 +2541,7 @@ _time:5m | unpack_logfmt fields (ip, host) skip_empty_results Performance tip: if you need extracting a single field from long [logfmt](https://brandur.org/logfmt) line, it is faster to use [`extract` pipe](#extract-pipe). For example, the following query extracts `"ip"` field from [logfmt](https://brandur.org/logfmt) line stored -in [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field): +in [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): ``` _time:5m | extract ' ip=' @@ -2558,7 +2558,7 @@ _time:5m | unpack_logfmt from foo result_prefix "foo_" Performance tips: - It is better from performance and resource usage PoV ingesting parsed [logfmt](https://brandur.org/logfmt) logs into VictoriaLogs - according to the [supported data model](./VictoriaLogs/keyConcepts.md#data-model) + according to the [supported data model](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) instead of ingesting unparsed logfmt lines into VictoriaLogs and then parsing them at query time with [`unpack_logfmt` pipe](#unpack_logfmt-pipe). - It is recommended using more specific [log filters](#filters) in order to reduce the number of log entries, which are passed to `unpack_logfmt`. @@ -2573,7 +2573,7 @@ See also: #### Conditional unpack_logfmt -If the [`unpack_logfmt` pipe](#unpack_logfmt-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`unpack_logfmt` pipe](#unpack_logfmt-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `unpack_logfmt`. The `` can contain arbitrary [filters](#filters). For example, the following query unpacks logfmt fields from `foo` field only if `ip` field in the current log entry isn't set or empty: @@ -2585,7 +2585,7 @@ _time:5m | unpack_logfmt if (ip:"") from foo ### unpack_syslog pipe `| unpack_syslog from field_name` [pipe](#pipes) unpacks [syslog](https://en.wikipedia.org/wiki/Syslog) message -from the given [`field_name`](./VictoriaLogs/keyConcepts.md#data-model). It understands the following Syslog formats: +from the given [`field_name`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). It understands the following Syslog formats: - [RFC3164](https://datatracker.ietf.org/doc/html/rfc3164) aka `MMM DD hh:mm:ss HOSTNAME APP-NAME[PROCID]: MESSAGE` - [RFC5424](https://datatracker.ietf.org/doc/html/rfc5424) aka `1 TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE` @@ -2609,7 +2609,7 @@ The `` part is optional. If it is missing, then `priority`, `facility` and The `[STRUCTURED-DATA]` is parsed into fields with the `SD-ID.param1`, `SD-ID.param2`, ..., `SD-ID.paramN` names and the corresponding values according to [the specification](https://datatracker.ietf.org/doc/html/rfc5424#section-6.3). -For example, the following query unpacks [syslog](https://en.wikipedia.org/wiki/Syslog) message from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query unpacks [syslog](https://en.wikipedia.org/wiki/Syslog) message from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) across logs for the last 5 minutes: ```logsql @@ -2617,7 +2617,7 @@ _time:5m | unpack_syslog from _msg ``` The `from _msg` part can be omitted when [syslog](https://en.wikipedia.org/wiki/Syslog) message is unpacked -from the [`_msg` field](./VictoriaLogs/keyConcepts.md#message-field). +from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). The following query is equivalent to the previous one: ```logsql @@ -2648,7 +2648,7 @@ _time:5m | unpack_syslog from foo result_prefix "foo_" Performance tips: - It is better from performance and resource usage PoV ingesting parsed [syslog](https://en.wikipedia.org/wiki/Syslog) messages into VictoriaLogs - according to the [supported data model](./VictoriaLogs/keyConcepts.md#data-model) + according to the [supported data model](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) instead of ingesting unparsed syslog lines into VictoriaLogs and then parsing them at query time with [`unpack_syslog` pipe](#unpack_syslog-pipe). - It is recommended using more specific [log filters](#filters) in order to reduce the number of log entries, which are passed to `unpack_syslog`. @@ -2663,7 +2663,7 @@ See also: #### Conditional unpack_syslog -If the [`unpack_syslog` pipe](#unpack_syslog-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`unpack_syslog` pipe](#unpack_syslog-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `unpack_syslog`. The `` can contain arbitrary [filters](#filters). For example, the following query unpacks syslog message fields from `foo` field only if `hostname` field in the current log entry isn't set or empty: @@ -2675,9 +2675,9 @@ _time:5m | unpack_syslog if (hostname:"") from foo ### unroll pipe `| unroll by (field1, ..., fieldN)` [pipe](#pipes) can be used for unrolling JSON arrays from `field1`, `fieldN` -[log fields](./VictoriaLogs/keyConcepts.md#data-model) into separate rows. +[log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) into separate rows. -For example, the following query unrolls `timestamp` and `value` [log fields](./VictoriaLogs/keyConcepts.md#data-model) from logs for the last 5 minutes: +For example, the following query unrolls `timestamp` and `value` [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) from logs for the last 5 minutes: ```logsql _time:5m | unroll (timestamp, value) @@ -2692,7 +2692,7 @@ See also: #### Conditional unroll -If the [`unroll` pipe](#unpack_logfmt-pipe) mustn't be applied to every [log entry](./VictoriaLogs/keyConcepts.md#data-model), +If the [`unroll` pipe](#unpack_logfmt-pipe) mustn't be applied to every [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), then add `if ()` after `unroll`. The `` can contain arbitrary [filters](#filters). For example, the following query unrolls `value` field only if `value_type` field equals to `json_array`: @@ -2704,29 +2704,29 @@ _time:5m | unroll if (value_type:="json_array") (value) LogsQL supports the following functions for [`stats` pipe](#stats-pipe): -- [`avg`](#avg-stats) returns the average value over the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`avg`](#avg-stats) returns the average value over the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - [`count`](#count-stats) returns the number of log entries. -- [`count_empty`](#count_empty-stats) returns the number logs with empty [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`count_uniq`](#count_uniq-stats) returns the number of unique non-empty values for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`max`](#max-stats) returns the maximum value over the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`median`](#median-stats) returns the [median](https://en.wikipedia.org/wiki/Median) value over the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`min`](#min-stats) returns the minumum value over the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`quantile`](#quantile-stats) returns the given quantile for the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`row_any`](#row_any-stats) returns a sample [log entry](./VictoriaLogs/keyConcepts.md#data-model) per each selected [stats group](#stats-by-fields). -- [`row_max`](#row_max-stats) returns the [log entry](./VictoriaLogs/keyConcepts.md#data-model) with the minimum value at the given field. -- [`row_min`](#row_min-stats) returns the [log entry](./VictoriaLogs/keyConcepts.md#data-model) with the maximum value at the given field. -- [`sum`](#sum-stats) returns the sum for the given numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`sum_len`](#sum_len-stats) returns the sum of lengths for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`uniq_values`](#uniq_values-stats) returns unique non-empty values for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). -- [`values`](#values-stats) returns all the values for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). +- [`count_empty`](#count_empty-stats) returns the number logs with empty [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`count_uniq`](#count_uniq-stats) returns the number of unique non-empty values for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`max`](#max-stats) returns the maximum value over the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`median`](#median-stats) returns the [median](https://en.wikipedia.org/wiki/Median) value over the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`min`](#min-stats) returns the minumum value over the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`quantile`](#quantile-stats) returns the given quantile for the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`row_any`](#row_any-stats) returns a sample [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) per each selected [stats group](#stats-by-fields). +- [`row_max`](#row_max-stats) returns the [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the minimum value at the given field. +- [`row_min`](#row_min-stats) returns the [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the maximum value at the given field. +- [`sum`](#sum-stats) returns the sum for the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`sum_len`](#sum_len-stats) returns the sum of lengths for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`uniq_values`](#uniq_values-stats) returns unique non-empty values for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). +- [`values`](#values-stats) returns all the values for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). ### avg stats `avg(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the average value across -all the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +all the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). Non-numeric values are ignored. -For example, the following query returns the average value for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the average value for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2752,7 +2752,7 @@ For example, the following query returns the number of logs over the last 5 minu _time:5m | stats count() logs ``` -It is possible calculating the number of logs with non-empty values for some [log field](./VictoriaLogs/keyConcepts.md#data-model) +It is possible calculating the number of logs with non-empty values for some [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the `count(fieldName)` syntax. For example, the following query returns the number of logs with non-empty `username` field over the last 5 minutes: ```logsql @@ -2760,7 +2760,7 @@ _time:5m | stats count(username) logs_with_username ``` If multiple fields are enumerated inside `count()`, then it counts the number of logs with at least a single non-empty field mentioned inside `count()`. -For example, the following query returns the number of logs with non-empty `username` or `password` [fields](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the number of logs with non-empty `username` or `password` [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over the last 5 minutes: ```logsql @@ -2778,7 +2778,7 @@ See also: `count_empty(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the number of logs with empty `(field1, ..., fieldN)` tuples. -For example, the following query calculates the number of logs with empty `username` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query calculates the number of logs with empty `username` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) during the last 5 minutes: ```logsql @@ -2794,14 +2794,14 @@ See also: `count_uniq(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the number of unique non-empty `(field1, ..., fieldN)` tuples. -For example, the following query returns the number of unique non-empty values for `ip` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the number of unique non-empty values for `ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over the last 5 minutes: ```logsql _time:5m | stats count_uniq(ip) ips ``` -The following query returns the number of unique `(host, path)` pairs for the corresponding [fields](./VictoriaLogs/keyConcepts.md#data-model) +The following query returns the number of unique `(host, path)` pairs for the corresponding [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over the last 5 minutes: ```logsql @@ -2825,9 +2825,9 @@ See also: ### max stats `max(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) returns the maximum value across -all the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +all the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query returns the maximum value for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the maximum value for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2846,9 +2846,9 @@ See also: ### median stats `median(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the [median](https://en.wikipedia.org/wiki/Median) value across -the give numeric [log fields](./VictoriaLogs/keyConcepts.md#data-model). +the give numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query return median for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query return median for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2863,9 +2863,9 @@ See also: ### min stats `min(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) returns the minimum value across -all the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +all the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query returns the minimum value for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the minimum value for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2884,10 +2884,10 @@ See also: ### quantile stats `quantile(phi, field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates `phi` [percentile](https://en.wikipedia.org/wiki/Percentile) over numeric values -for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). The `phi` must be in the range `0 ... 1`, where `0` means `0th` percentile, +for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). The `phi` must be in the range `0 ... 1`, where `0` means `0th` percentile, while `1` means `100th` percentile. -For example, the following query calculates `50th`, `90th` and `99th` percentiles for the `request_duration_seconds` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query calculates `50th`, `90th` and `99th` percentiles for the `request_duration_seconds` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -2906,10 +2906,10 @@ See also: ### row_any stats -`row_any()` [stats pipe function](#stats-pipe-functions) returns arbitrary [log entry](./VictoriaLogs/keyConcepts.md#data-model) +`row_any()` [stats pipe function](#stats-pipe-functions) returns arbitrary [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) (aka sample) per each selected [stats group](#stats-by-fields). Log entry is returned as JSON-encoded dictionary with all the fields from the original log. -For example, the following query returns a sample log entry per each [`_stream`](./VictoriaLogs/keyConcepts.md#stream-fields) +For example, the following query returns a sample log entry per each [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) across logs for the last 5 minutes: ```logsql @@ -2932,10 +2932,10 @@ See also: ### row_max stats -`row_max(field)` [stats pipe function](#stats-pipe-functions) returns [log entry](./VictoriaLogs/keyConcepts.md#data-model) +`row_max(field)` [stats pipe function](#stats-pipe-functions) returns [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the maximum value for the given `field`. Log entry is returned as JSON-encoded dictionary with all the fields from the original log. -For example, the following query returns log entry with the maximum value for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns log entry with the maximum value for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) across logs for the last 5 minutes: ```logsql @@ -2959,10 +2959,10 @@ See also: ### row_min stats -`row_min(field)` [stats pipe function](#stats-pipe-functions) returns [log entry](./VictoriaLogs/keyConcepts.md#data-model) +`row_min(field)` [stats pipe function](#stats-pipe-functions) returns [log entry](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) with the minimum value for the given `field`. Log entry is returned as JSON-encoded dictionary with all the fields from the original log. -For example, the following query returns log entry with the minimum value for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns log entry with the minimum value for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) across logs for the last 5 minutes: ```logsql @@ -2987,9 +2987,9 @@ See also: ### sum stats `sum(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the sum of numeric values across -all the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +all the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query returns the sum of numeric values for the `duration` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns the sum of numeric values for the `duration` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -3006,9 +3006,9 @@ See also: ### sum_len stats `sum_len(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) calculates the sum of lengths of all the values -for the given [log fields](./VictoriaLogs/keyConcepts.md#data-model). +for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). -For example, the following query returns the sum of lengths of [`_msg` fields](./VictoriaLogs/keyConcepts.md#message-field) +For example, the following query returns the sum of lengths of [`_msg` fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) across all the logs for the last 5 minutes: ```logsql @@ -3022,10 +3022,10 @@ See also: ### uniq_values stats `uniq_values(field1, ..., fieldN)` [stats pipe function](#stats-pipe-functions) returns the unique non-empty values across -the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). The returned values are encoded in sorted JSON array. -For example, the following query returns unique non-empty values for the `ip` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns unique non-empty values for the `ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -3037,7 +3037,7 @@ The returned unique ip addresses can be unrolled into distinct log entries with Every unique value is stored in memory during query execution. Big number of unique values may require a lot of memory. Sometimes it is enough to return only a subset of unique values. In this case add `limit N` after `uniq_values(...)` in order to limit the number of returned unique values to `N`, while limiting the maximum memory usage. -For example, the following query returns up to `100` unique values for the `ip` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns up to `100` unique values for the `ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over the logs for the last 5 minutes: ```logsql @@ -3056,10 +3056,10 @@ See also: ### values stats `values(field1, ..., fieldN)` [stats pipe fuction](#stats-pipe-functions) returns all the values (including empty values) -for the mentioned [log fields](./VictoriaLogs/keyConcepts.md#data-model). +for the mentioned [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). The returned values are encoded in JSON array. -For example, the following query returns all the values for the `ip` [field](./VictoriaLogs/keyConcepts.md#data-model) +For example, the following query returns all the values for the `ip` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) over logs for the last 5 minutes: ```logsql @@ -3082,32 +3082,32 @@ See [`stream_context` pipe](#stream_context-pipe). LogsQL supports the following transformations on the log entries selected with [filters](#filters): -- Extracting arbitrary text from [log fields](./VictoriaLogs/keyConcepts.md#data-model) according to the provided pattern. +- Extracting arbitrary text from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) according to the provided pattern. See [these docs](#extract-pipe) for details. -- Unpacking JSON fields from [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](#unpack_json-pipe). -- Unpacking [logfmt](https://brandur.org/logfmt) fields from [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](#unpack_logfmt-pipe). -- Unpacking [Syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [these docs](#unpack_syslog-pipe). -- Creating a new field from existing [log fields](./VictoriaLogs/keyConcepts.md#data-model) according to the provided format. See [`format` pipe](#format-pipe). -- Replacing substrings in the given [log field](./VictoriaLogs/keyConcepts.md#data-model). +- Unpacking JSON fields from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](#unpack_json-pipe). +- Unpacking [logfmt](https://brandur.org/logfmt) fields from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](#unpack_logfmt-pipe). +- Unpacking [Syslog](https://en.wikipedia.org/wiki/Syslog) messages from [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [these docs](#unpack_syslog-pipe). +- Creating a new field from existing [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) according to the provided format. See [`format` pipe](#format-pipe). +- Replacing substrings in the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [`replace` pipe](#replace-pipe) and [`replace_regexp` pipe](#replace_regexp-pipe) docs. -- Creating a new field according to math calculations over existing [log fields](./VictoriaLogs/keyConcepts.md#data-model). See [`math` pipe](#math-pipe). +- Creating a new field according to math calculations over existing [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [`math` pipe](#math-pipe). It is also possible to perform various transformations on the [selected log entries](#filters) at client side -with `jq`, `awk`, `cut`, etc. Unix commands according to [these docs](./VictoriaLogs/querying/README.md#command-line). +with `jq`, `awk`, `cut`, etc. Unix commands according to [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line). ## Post-filters Post-filtering of query results can be performed at any step by using [`filter` pipe](#filter-pipe). It is also possible to perform post-filtering of the [selected log entries](#filters) at client side with `grep` and similar Unix commands -according to [these docs](./VictoriaLogs/querying/README.md#command-line). +according to [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line). ## Stats Stats over the selected logs can be calculated via [`stats` pipe](#stats-pipe). It is also possible to perform stats calculations on the [selected log entries](#filters) at client side with `sort`, `uniq`, etc. Unix commands -according to [these docs](./VictoriaLogs/querying/README.md#command-line). +according to [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line). ## Sorting @@ -3117,7 +3117,7 @@ By default VictoriaLogs doesn't sort the returned results because of performance LogsQL provides the following [pipes](#pipes) for limiting the number of returned log entries: -- [`fields`](#fields-pipe) and [`delete`](#delete-pipe) pipes allow limiting the set of [log fields](./VictoriaLogs/keyConcepts.md#data-model) to return. +- [`fields`](#fields-pipe) and [`delete`](#delete-pipe) pipes allow limiting the set of [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) to return. - [`limit` pipe](#limit-pipe) allows limiting the number of log entries to return. ## Querying specific fields @@ -3183,7 +3183,7 @@ Internally duration values are converted into nanoseconds. - It is highly recommended specifying [time filter](#time-filter) in order to narrow down the search to specific time range. - It is highly recommended specifying [stream filter](#stream-filter) in order to narrow down the search - to specific [log streams](./VictoriaLogs/keyConcepts.md#stream-fields). + to specific [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). - Move faster filters such as [word filter](#word-filter) and [phrase filter](#phrase-filter) to the beginning of the query. This rule doesn't apply to [time filter](#time-filter) and [stream filter](#stream-filter), which can be put at any place of the query. - Move more specific filters, which match lower number of log entries, to the beginning of the query. diff --git a/docs/data-ingestion/README.md b/docs/data-ingestion/README.md index 325c49165..58b9ed793 100644 --- a/docs/data-ingestion/README.md +++ b/docs/data-ingestion/README.md @@ -1,5 +1,5 @@ In This Folder you will find instructions for sending data to VictoriaMetrics from a variety of platforms. -If your tool is not listed it is likely you can ingest your data into VictoriaMetrics using one of the protocols listed in our [Prominent features](../Single-server-VictoriaMetrics.md#prominent-features") section. +If your tool is not listed it is likely you can ingest your data into VictoriaMetrics using one of the protocols listed in our [Prominent features]({{< ref "/Single-server-VictoriaMetrics.md#prominent-features" >}}) section. If you are unsure what port number to use when pushing data to VictoriaMetrics single node, vminsert, vmagent, and vmauth we have listed the default ports below. @@ -11,9 +11,9 @@ If you are unsure what port number to use when pushing data to VictoriaMetrics s In the rest of the documentation we will assume you have configured your push endpoint to use TLS/SSL on port 443 so the urls in the rest of the documentation will look like `https://` instead of `http://:8428` for VictoriaMetrics single. ## Documented Collectors/Agents -* [Telegraf](./Telegraf.md) -* [Vector](./Vector.md) +* [Telegraf]({{< relref "Telegraf.md" >}}) +* [Vector]({{< relref "Vector.md" >}}) ## Supported Platforms -* [Proxmox Virtual Environment and Proxmox Backup Server](./Proxmox.md) +* [Proxmox Virtual Environment and Proxmox Backup Server]({{< relref "Proxmox.md" >}}) diff --git a/docs/data-ingestion/Vector.md b/docs/data-ingestion/Vector.md index 304a08633..47489111e 100644 --- a/docs/data-ingestion/Vector.md +++ b/docs/data-ingestion/Vector.md @@ -117,4 +117,4 @@ sinks: # References - [Vector documentation](https://vector.dev/docs/) -- [VictoriaLogs documentation for using vector](../VictoriaLogs/data-ingestion/Vector.md) +- [VictoriaLogs documentation for using vector]({{< ref "/victorialogs/data-ingestion/vector" >}})