Document the solution - to switch to eval_delay option at group config.
This is a follow-up for e80b44f19d
100 KiB
sort | weight | title | menu | aliases | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
100 | 100 | CHANGELOG |
|
|
CHANGELOG
The following tip
changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:
- How to build single-node VictoriaMetrics
- How to build cluster version of VictoriaMetrics
- How to build vmagent
- How to build vmalert
- How to build vmauth
- How to build vmctl
Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation runs under the constant load generated by prometheus-benchmark and is used for testing latest releases.
See also LTS releases.
tip
Update note 1: the -datasource.lookback
command-line flag at vmalert
is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay
option. See this issue for more details.
-
SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.1. See the list of issues addressed in Go1.22.1.
-
FEATURE: vmauth: allow discovering ip addresses for backend instances hidden behind a shared hostname, via
discover_backend_ips: true
option. This allows evenly spreading load among backend instances. See these docs and this feature request. -
FEATURE: vmauth: allow routing incoming requests based on HTTP query args via
src_query_args
option aturl_map
. See these docs and this feature request. -
FEATURE: vmauth: allow routing incoming requests based on HTTP request headers via
src_headers
option aturl_map
. See these docs. -
FEATURE: stream aggregation: reduce memory usage by up to 5x when aggregating over big number of unique time series. The memory usage reduction is most visible when stream deduplication is enabled.
-
FEATURE: stream aggregation: allow using
-streamAggr.dedupInterval
and-remoteWrite.streamAggr.dedupInterval
command-line flags without the need to specify-streamAggr.config
and-remoteWrite.streamAggr.config
. See these docs. -
FEATURE: stream aggregation: add
-streamAggr.dropInputLabels
command-line flag, which can be used for dropping the listed labels from input samples before applying stream de-duplication and aggregation. This is faster and easier to use alternative to input_relabel_configs. See these docs. -
FEATURE: stream aggregation: add
dedup_interval
option, which allows configuring individual deduplication intervals per each stream aggregation config. -
FEATURE: stream aggregation: add
keep_metric_names
option, which can be set at stream aggregation config in order to keep the original metric names in the output aggregated samples instead of using the default output metric naming scheme. -
FEATURE: stream aggregation: align the time of aggregated data flush to the specified aggregation
interval
. For example, ifinterval
is set to1m
, then the aggregated data will be flushed at the end of every minute. The alginment can be disabled by settingno_align_flush_to_interval: true
option at stream aggregation config. See these docs for details. -
FEATURE: stream aggregation: add unique_samples output, which can be used for calculating the number of unique sample values over the given
interval
. -
FEATURE: stream aggregation: add increase_prometheus and total_prometheus outputs, which can be used for
increase
andtotal
aggregations when the first sample of every new time series must be ignored. -
FEATURE: stream aggregation: expose
vm_streamaggr_flush_timeouts_total
andvm_streamaggr_dedup_flush_timeouts_total
counters at/metrics
page, which can be used for detecting flush timeouts for stream aggregation states. Expose alsovm_streamaggr_flush_duration_seconds
andvm_streamaggr_dedup_flush_duration_seconds
histograms for monitoring the real flush durations of stream aggregation states. -
FEATURE: vmui: improve trace display for better visual separation of branches. See this pull request.
-
FEATURE: vmagent: use the provided
-remoteWrite.tlsServerName
asHost
header in requests to-remoteWrite.url
. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and the pull request. -
FEATURE: optimize
/api/v1/labels
and/api/v1/label/.../values
whenmatch[]
filters contains metric name. For example,/api/v1/label/instance/values?match[]=up
now works much faster than before. See this issue. -
FEATURE: vmctl: support client-side TLS configuration for native protocol. See this feature request. Thanks to @khushijain21 for the pull request.
-
FEATURE: vmctl: support client-side TLS configuration for VictoriaMetrics destination specified via
--vm-*
cmd-line flags used in InfluxDB, Remote Read protocol, OpenTSDB, Prometheus and Promscale migration modes. -
BUGFIX: vmagent: properly set
Host
header in requests to scrape targets if it is specified viaheaders
option. Thanks to @fholzer for the bugreport and the fix. -
BUGFIX: vmagent: properly set
Host
header in requests to scrape targets whenserver_name
option attls_config
is set. Previously theHost
header was set incorrectly to the target hostname in this case. -
BUGFIX: do not drop
match[]
filter at/api/v1/series
if-search.ignoreExtraFiltersAtLabelsAPI
command-line flag is set, since missingmatch[]
filter breaks/api/v1/series
requests. -
BUGFIX: stream aggregation: take into account sample timestamp during deduplication. If multiple samples have the same timestamp on the configured deduplication interval, then the sample with the biggest value is kept. The logic is aligned with
-dedup.minScrapeInterval
. -
BUGFIX: vmctl: properly parse TLS key and CA files for InfluxDB and OpenTSDB migration modes.
v1.99.0
12/03/2024: There's ongoing investigation of the issue #5959 related to this release. Until the issue is addressed, please avoid updating to v1.99.0.
Released at 2024-03-01
-
FEATURE: MetricsQL: propagate label filters via all the label manipulation functions. For example,
label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"}
is now transformed tolabel_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}
. This should reduce the amounts of time series processed during query execution. -
FEATURE: MetricsQL: add count_values_over_time function. See this feature request.
-
FEATURE: Single-node VictoriaMetrics and
vmstorage
in VictoriaMetrics cluster: exposevm_last_partition_parts
metrics, which show the number of parts in the latest partition. These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition. -
FEATURE: Single-node VictoriaMetrics and
vmstorage
in VictoriaMetrics cluster: exposevm_snapshots
metric, which shows the current number of snapshots created via snapshot API. -
FEATURE: Single-node VictoriaMetrics and
vmselect
in VictoriaMetrics cluster: add-search.ignoreExtraFiltersAtLabelsAPI
command-line flag, which can be used for reducing load on VictoriaMetrics when /api/v1/labels, /api/v1/label/.../values or /api/v1/series are queried with too broadextra_filters
orextra_label
, which match many time series. See these docs for details. -
FEATURE: Single-node VictoriaMetrics and
vmselect
in VictoriaMetrics cluster: allow limiting CPU and RAM usage at /api/v1/labels, /api/v1/label/.../values and /api/v1/series on systems with high churn rate via-search.maxLabelsAPIDuration
and-search.maxLabelsAPISeries
command-line flags. See these docs for details. -
FEATURE: vmagent: support reading Amazon CloudWatch metrics in OpenTelemetry format from Amazon Data Firehose.
-
FEATURE: vmagent: add support for
client_id
option into kuma_sd_configs in the same way as Prometheus does. See this pull request. -
FEATURE: vmagent: add support for
enable_compression
option in scrape_configs in order to be compatible with Prometheus scrape configs. See this pull request and this feature request. Note thatvmagent
was always supportingdisable_compression
option before Prometheus addedenable_compression
option. -
FEATURE: vmctl: support client-side TLS configuration for InfluxDB, Remote Read protocol and OpenTSDB. See this feature request. Thanks to @khushijain21 for pull requests 1, 2, 3.
-
FEATURE: vmui: preserve
WITH
templates when clicking theprettify query
button at the right side of query input field. See this feature request. -
FEATURE: vmui: allow filling gaps on graphs with interpolated lines as Grafana does. See this feature request and this pull request.
-
FEATURE: vmalert: support filtering by group, rule or labels in vmalert's UI for
/groups
and/alerts
pages. See the pull request by @victoramsantos. -
FEATURE: docker-compose: create a separate docker-compose environment for VictoriaLogs installation, including fluentbit and VictoriaLogs Grafana datasource. See these docs for details.
-
FEATURE: vmbackupmanager: wait for up 30 seconds before making a snapshot for backup if
vmstorage
is temporarily unavailalbe. This should prevent fromvmbackupmanager
termination in this case. See this feature request. -
BUGFIX: downgrade Go builder from
1.22.0
to1.21.7
, since1.22.0
contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. -
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to
-httpListenAddr
every 2 minutes, which was disabled in v1.98.0 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back. -
BUGFIX: vmagent: accept OpenTelemetry data at
/opentelemetry/v1/metrics
, since the/v1/metrics
suffix is hardcoded at OpenTelemetry protocol specification. -
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with
-remoteWrite.shardByURL
command-line flag. Thanks to @penguinlav for the fix for this issue. -
BUGFIX: fix the misleading error
0ms is out of allowed range [0 ...
when passingstep=0
to /api/v1/query or /api/v1/query_range. See this issue. -
BUGFIX: Single-node VictoriaMetrics and
vmselect
in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details. -
BUGFIX: vmalert: consistently sort groups by name and filename on
/groups
page in UI. This should prevent non-deterministic sorting for groups with identical names. -
BUGFIX: vmui: fix Popper display issues. See this pull request.
-
BUGFIX: vmselect: format time with milliseconds precision when
__timestamp__:rfc3339
is specified for /api/v1/export/csv API response. See this issue for details. -
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.
v1.98.0
Released at 2024-02-14
-
SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
-
FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at
-httpListenAddr
(aka mTLS). See these docs and this feature request. -
FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple
-httpListenAddr
command-line flags. For example,./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428
starts VictoriaMetrics, which accepts incoming http requests at bothsome-host:12345
andlocalhost:8428
. See this feature request. -
FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example,
-remoteWrite.sendTimeout=',20s,'
specifies three-remoteWrite.sendTimeout
values - the first one and the last one have default values (30s
in this case), while the second one is set to20s
. -
FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.
-
FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka
/api/beta/sketches
endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request. -
FEATURE: vmagent: add
-remoteWrite.tlsHandshakeTimeout
command-line flag for tuning the timeout needed for establishing TLS connections to-remoteWrite.url
. Bigger tls handshake timeouts should reduce the probability ofhttp: TLS handshake error from ...: EOF
errors at the remote storage side under high load. See this issue. -
FEATURE: VictoriaMetrics cluster: add
-disableReroutingOnUnavailable
command-line flag tovminsert
, which can be used for reducing resource usage spikes atvmstorage
nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request. -
FEATURE: add
-search.resetRollupResultCacheOnStartup
command-line flag for resetting query cache on startup. See this feature request. -
FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example,
label_set(q1, "a", "b") + q2{c="d"}
is automatically transformed tolabel_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"}
now. This should improve performance for such queries. See this issue. -
FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.
-
FEATURE: dashboards/vmagent: add
Targets scraped/s
stat panel showing the number of targets scraped by the vmagent per-second. -
FEATURE: dashboards/all: add new panel
CPU spent on GC
. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed. -
FEATURE: vmalert: support filtering for
/api/v1/rules
API. See the pull request by @victoramsantos. -
FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via
-snapshot.tls*
cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request. -
FEATURE: vmui: add a time picker to the "Logs Explorer" page. See this issue.
-
BUGFIX: all VictoriaMetrics components: do not close connections to
-httpListenAddr
every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to-http.connTimeout
command-line flag. -
BUGFIX: vmagent: reduce CPU usage when
-promscrape.dropOriginalLabels
command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request. -
BUGFIX: vmauth: properly release memory during config reload. See this issue.
-
BUGFIX: vmauth: properly expose
vmauth_unauthorized_user_concurrent_requests_capacity
,vmauth_unauthorized_user_concurrent_requests_current
,vmauth_user_concurrent_requests_capacity
andvmauth_user_concurrent_requests_current
metrics after config reload. Previously these metrics didn't work after config reload. -
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example,
sum({job="foo"}, {job="bar"}) by (job) + a
was improperly optimized tosum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}
before being executed. This could lead to unexpected results. See this issue. -
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
-
BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from
/-/reload
endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request. -
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
-
BUGFIX: prevent from possible
too big indexBlockSize
panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics. -
BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.
-
BUGFIX: vmui: fix handling invalid timezone. See this issue.
-
BUGFIX: vmui: fix the bug where the select does not open. See this issue.
-
BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.
-
BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.
-
BUGFIX: dashboards: update
Storage full ETA
panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request. -
BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.
v1.97.3
Released at 2024-03-01
v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release
- BUGFIX: downgrade Go builder from
1.22.0
to1.21.7
, since1.22.0
contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. - BUGFIX: take into account
-search.maxUniqueTimeseries
limit at /api/v1/labels, /api/v1/label/.../values and /api/v1/series. This limit was ignored at these endpoints starting from v1.97.0 when trying to address this issue. It has been appeared that this can result in high CPU usage when heaviweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to returnlimit exceeded
error to heavyweight queries instead of burning CPU. - BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to
-httpListenAddr
every 2 minutes, which was disabled in v1.97.2 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back. - BUGFIX: vmagent: accept OpenTelemetry data at
/opentelemetry/v1/metrics
, since the/v1/metrics
suffix is hardcoded at OpenTelemetry protocol specification. - BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with
-remoteWrite.shardByURL
command-line flag. Thanks to @penguinlav for the fix for this issue. - BUGFIX: fix the misleading error
0ms is out of allowed range [0 ...
when passingstep=0
to /api/v1/query or /api/v1/query_range. See this issue. - BUGFIX: Single-node VictoriaMetrics and
vmselect
in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details. - BUGFIX: vmalert: consistently sort groups by name and filename on
/groups
page in UI. This should prevent non-deterministic sorting for groups with identical names. - BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.
- BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.
v1.97.2
Released at 2024-02-14
v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release
-
SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
-
BUGFIX: all VictoriaMetrics components: do not close connections to
-httpListenAddr
every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to-http.connTimeout
command-line flag. -
BUGFIX: vmagent: reduce CPU usage when
-promscrape.dropOriginalLabels
command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request. -
BUGFIX: vmauth: properly release memory during config reload. See this issue.
-
BUGFIX: vmauth: properly expose
vmauth_unauthorized_user_concurrent_requests_capacity
,vmauth_unauthorized_user_concurrent_requests_current
,vmauth_user_concurrent_requests_capacity
andvmauth_user_concurrent_requests_current
metrics after config reload. Previously these metrics didn't work after config reload. -
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example,
sum({job="foo"}, {job="bar"}) by (job) + a
was improperly optimized tosum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}
before being executed. This could lead to unexpected results. See this issue. -
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
-
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
-
BUGFIX: prevent from possible
too big indexBlockSize
panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
v1.97.1
Released at 2024-02-01
v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release
-
FEATURE: vmagent: add support for
username_file
option atbasic_auth
section inscrape_configs
. See this feature request. Thanks to @wasim-nihal for the initial implementation. -
BUGFIX: vmagent: fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in v1.97.0.
-
BUGFIX: fix
runtime error: slice bounds out of range
panic, which can occur during query execution. See this issue. The bug has been introduced inv1.97.0
. -
BUGFIX: MetricsQL: properly handle
avg_over_time({some_filter}[d]) keep_metric_names
queries, wheresome_filter
matches multiple time series with multiple names, whiled
is bigger or equal to3h
. See this issue. -
BUGFIX: dashboards/single: fix typo in query for
version
annotation which falsely produced many version change events.
v1.97.0
Released at 2024-01-30
v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release
-
SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
-
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
-
FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.
-
FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.
-
FEATURE: graphite: add support for negative index in
groupByNode
andaliasByNode
functions. Thanks to @rbizos for the pull request. -
FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.
-
FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.
-
FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.
-
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each
-remoteWrite.url
via the command-line flag-remoteWrite.oauth2.endpointParams
. See these docs. Thanks to @mhill-holoplot for the pull request. -
FEATURE: vmagent: add ability to set
attach_metadata.node=true
option for all thekubernetes_sd_configs
defined at-promscrape.config
via-promscrape.kubernetes.attachNodeMetadataAll
command-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation. -
FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying
flush_on_shutdown: true
setting in streaming aggregation config. See more details here. -
FEATURE: streaming aggregation: expand
%{ENV_VAR}
placeholders in config files with the corresponding environment variable values. -
FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
-datasource.oauth2.endpointParams
for-datasource.url
-notifier.oauth2.endpointParams
for-notifier.url
-remoteRead.oauth2.endpointParams
for-remoteRead.url
-remoteWrite.oauth2.endpointParams
for-remoteWrite.url
-
FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via
src_hosts
option aturl_map
. See these docs. -
FEATURE: vmauth: expose
vmauth_user_request_backend_errors_total
andvmauth_unauthorized_user_request_backend_errors_total
metrics, which track the number of failed requests because of backend errors. See this feature request. -
FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via
metric_labels
section. See this feature request. -
FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at
-httpListenAddr
already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue. -
FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to
-pushmetrics.url
. Seepush metrics
docs. -
FEATURE: all VictoriaMetrics components: add
-metrics.exposeMetadata
command-line flag, which allows displayingTYPE
andHELP
metadata at/metrics
page exposed at-httpListenAddr
. This may be needed when the/metrics
page is scraped by collector, which requires theTYPE
andHELP
metadata such as Google Cloud Managed Prometheus. -
FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using
file:///path/to/file
orhttp://host/path
syntax for the following command-line flags:-configAuthKey
,-deleteAuthKey
,-flagsAuthKey
,-forceMergeAuthKey
,-forceFlushAuthKey
,-httpAuth.password
,-metricsAuthKey
,-pprofAuthKey
,-reloadAuthKey
,-search.resetCacheAuthKey
,-snapshotAuthKey
. For example,-httpAuth.password=file:///path/to/password
. See these docs for details. -
FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
-
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
-
FEATURE: dashboards/single: change dashboard title from
VictoriaMetrics
toVictoriaMetrics - single-node
. The new title should provide better understanding of this dashboard purpose. -
FEATURE: vmctl: rename cmd-line flag
vm-native-disable-retries
tovm-native-disable-per-metric-migration
to better reflect its meaning. -
FEATURE: vmctl: add
-vm-native-src-insecure-skip-verify
and-vm-native-dst-insecure-skip-verify
command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses. -
FEATURE: Alerting rules for VictoriaMetrics: add
job
label toDiskRunsOutOfSpace
alerting rule, so it is easier to understand to which installation the triggered instance belongs. -
FEATURE:
vmstorage
: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by-maxLabelsPerTimeseries
or-maxLabelValueLen
command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong. -
FEATURE: vmui: add the ability to export and import query reports:
- add a
Query Analyzer
page that allows you to build graphs fromJSON
data containing the results of executing a query request. - add an
Export query
button to the graph that saves the result of executing the query inJSON
. See this pull request.
- add a
-
FEATURE: vmui: add
-vmui.defaultTimezone
flag to set a default timezone. See this issue and these docs. -
FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.
-
FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.
-
FEATURE: add VictoriaMetrics datasource to docker compose environment. See this pull request.
-
BUGFIX: properly return the list of matching label names and label values from
/api/v1/labels
and/api/v1/label/.../values
when the database contains more than-search.maxUniqueTimeseries
unique time series on the selected time range. Previously VictoriaMetrics could returnthe number of matching timeseries exceeds ...
error in this case. See this issue. -
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
-
BUGFIX: VictoriaMetrics cluster: properly return full results when
-search.skipSlowReplicas
command-line flag is passed tovmselect
and when vmstorage groups are in use. Previously partial results could be returned in this case. -
BUGFIX:
vminsert
: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped. -
BUGFIX:
vmstorage
: added missing-inmemoryDataFlushInterval
command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0. -
BUGFIX:
vmstorage
: properly expirestorage/prefetchedMetricIDs
cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time. -
BUGFIX: vmalert: check
-external.url
schema when starting vmalert, must behttp
orhttps
. Before, alertmanager could reject alert notifications if-external.url
contained no or wrong schema. -
BUGFIX: vmalert: automatically add
exported_
prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue. -
BUGFIX: vmalert: autogenerate
ALERTS_FOR_STATE
time series for alerting rules withfor: 0
. Previously,ALERTS_FOR_STATE
was generated only for alerts withfor > 0
. The change aligns with Prometheus behavior. See more details in this issue. -
BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with
context cancelled
error. -
BUGFIX: MetricsQL: consistently sort results for
q1 or q2
query, so they do not change colors with each refresh in Grafana. See this issue. -
BUGFIX: MetricsQL: properly return results from bottomk and
bottomk_*()
functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix. -
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example,
sum(quantile_over_time(0.5, process_resident_memory_bytes))
was resulting toexpecting at least 2 args to ...; got 1 args
error. Thanks to @atykhyy for the pull request. -
BUGFIX: vmctl: retry on import errors in
vm-native
mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. -
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
-
BUGFIX: vmagent: exit if there is config syntax error in
scrape_config_files
when-promscrape.config.strictParse=true
. See this issue. -
BUGFIX: vmagent: properly discover targets for
role: endpoints
androle: endpointslice
in kubernetes_sd_configs. Previously someendpoints
andendpointslice
targets could be left undiscovered or some targets could have missing__meta_*
labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request. -
BUGFIX: vmagent: respect explicitly set
series_limit: 0
in scrape_config. This allows removingseries_limit
restriction on a per-scrape_config
basis when global limit is set via-promscrape.seriesLimitPerTarget
. Previously,0
value was ignored in favor of-promscrape.seriesLimitPerTarget
. -
BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in
kubernetes_sd_configs
. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them. -
BUGFIX: vmagent: properly follow redirects to another hostnames when
follow_redirects: true
option is set inscrape_configs
while stream parsing mode is disabled. See this issue and this pull request. -
BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.
-
BUGFIX: vmui: fix the display of autocomplete results and cache the results. See this issue and this issue.
-
BUGFIX: vmui: send
step
param for instant queries. The change reverts this issue due to reasons explained in this comment. -
BUGFIX: vmui: fix the
Enter
key press issue in theQuery
field when using autocomplete. See this issue. -
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
-
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as
foo[100y]
. Previously, such queries could return empty responses even iffoo
is present in database. See this issue. -
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
-
BUGFIX:
vmselect
: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached. -
BUGFIX:
vmselect
: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example,sum(avg_over_time(up[24h]))
. This bug has been introduced in v1.95.0. See this and this issues.
v1.96.0
See changes here
v1.95.1
See changes here
v1.95.0
See changes here
v1.94.0
See changes here
v1.93.13
Released at 2024-03-01
v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
-
BUGFIX: downgrade Go builder from
1.22.0
to1.21.7
, since1.22.0
contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage. -
BUGFIX: vmagent: accept OpenTelemetry data at
/opentelemetry/v1/metrics
, since the/v1/metrics
suffix is hardcoded at OpenTelemetry protocol specification. -
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with
-remoteWrite.shardByURL
command-line flag. Thanks to @penguinlav for the fix for this issue. -
BUGFIX: fix the misleading error
0ms is out of allowed range [0 ...
when passingstep=0
to /api/v1/query or /api/v1/query_range. See this issue. -
BUGFIX: Single-node VictoriaMetrics and
vmselect
in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details. -
BUGFIX: vmalert: consistently sort groups by name and filename on
/groups
page in UI. This should prevent non-deterministic sorting for groups with identical names. -
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.
v1.93.12
Released at 2024-02-14
v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
-
SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
-
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example,
sum({job="foo"}, {job="bar"}) by (job) + a
was improperly optimized tosum({job="foo"}, {job="bar"}) by (job) + a{job="foo"}
before being executed. This could lead to unexpected results. See this issue. -
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
-
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
-
BUGFIX: prevent from possible
too big indexBlockSize
panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
v1.93.11
Released at 2024-02-01
v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
-
SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
-
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
-
BUGFIX: vmagent: properly discover targets for
role: endpoints
androle: endpointslice
in kubernetes_sd_configs. Previously someendpoints
andendpointslice
targets could be left undiscovered or some targets could have missing__meta_*
labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request. -
BUGFIX: vmagent: respect explicitly set
series_limit: 0
in scrape_config. This allows removingseries_limit
restriction on a per-scrape_config
basis when global limit is set via-promscrape.seriesLimitPerTarget
. Previously,0
value was ignored in favor of-promscrape.seriesLimitPerTarget
. -
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as
foo[100y]
. Previously, such queries could return empty responses even iffoo
is present in database. See this issue. -
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
v1.93.10
Released at 2024-01-17
v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
-
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
-
BUGFIX:
vminsert
: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped. -
BUGFIX:
vmstorage
: added missing-inmemoryDataFlushInterval
command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0. -
BUGFIX:
vmstorage
: properly expirestorage/prefetchedMetricIDs
cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time. -
BUGFIX: vmalert: check
-external.url
schema when starting vmalert, must behttp
orhttps
. Before, alertmanager could reject alert notifications if-external.url
contained no or wrong schema. -
BUGFIX: MetricsQL: properly return results from bottomk and
bottomk_*()
functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix. -
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example,
sum(quantile_over_time(0.5, process_resident_memory_bytes))
was resulting toexpecting at least 2 args to ...; got 1 args
error. Thanks to @atykhyy for the pull request. -
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
-
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
-
BUGFIX: vmctl: check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the pull request.
-
BUGFIX: vmctl: retry on import errors in
vm-native
mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
v1.93.9
See changes here
v1.93.8
See changes here
v1.93.7
See changes here
v1.93.6
See changes here
v1.93.5
See changes here
v1.93.4
See changes here
v1.93.3
See changes here
v1.93.2
See changes here
v1.93.1
See changes here
v1.93.0
See changes here
v1.92.1
See changes here
v1.92.0
See changes here
v1.91.3
See changes here
v1.91.2
See changes here
v1.91.1
See changes here
v1.91.0
See changes here
v1.90.0
See changes here
v1.89.1
See changes here
v1.89.0
See changes here
v1.88.1
See changes here
v1.88.0
See changes here
v1.87.14
Released at 2024-02-01
v1.87.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
-
SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
-
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
-
BUGFIX: vmagent: respect explicitly set
series_limit: 0
in scrape_config. This allows removingseries_limit
restriction on a per-scrape_config
basis when global limit is set via-promscrape.seriesLimitPerTarget
. Previously,0
value was ignored in favor of-promscrape.seriesLimitPerTarget
. -
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as
foo[100y]
. Previously, such queries could return empty responses even iffoo
is present in database. See this issue. -
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
v1.87.13
Released at 2024-01-17
v1.87.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
-
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
-
BUGFIX:
vmstorage
: added missing-inmemoryDataFlushInterval
command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0. -
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example,
sum(quantile_over_time(0.5, process_resident_memory_bytes))
was resulting toexpecting at least 2 args to ...; got 1 args
error. Thanks to @atykhyy for the pull request. -
BUGFIX:
vmstorage
: properly expirestorage/prefetchedMetricIDs
cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time. -
BUGFIX: MetricsQL: properly return results from bottomk and
bottomk_...
functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix. -
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
v1.87.12
See changes here
v1.87.11
See changes here
v1.87.10
See changes here
v1.87.9
See changes here
v1.87.8
See changes here
v1.87.7
See changes here
v1.87.6
See changes here
v1.87.5
See changes here
v1.87.4
See changes here
v1.87.3
See changes here
v1.87.2
See changes here
v1.87.1
See changes here
v1.87.0
See changes here
v1.86.2
See changes here
v1.86.1
See changes here
v1.86.0
See changes here
v1.85.3
See changes here
v1.85.2
See changes here
v1.85.1
See changes here
v1.85.0
See changes here
v1.84.0
See changes here
v1.83.1
See changes here
v1.83.0
See changes here
v1.82.1
See changes here
v1.82.0
See changes here
v1.81.2
See changes here
v1.81.1
See changes here
v1.81.0
See changes here
v1.80.0
See changes here
v1.79.14
See changes here
v1.79.13
See changes here
v1.79.12
See changes here
v1.79.11
See changes here
v1.79.10
See changes here
v1.79.9
See changes here
v1.79.8
See changes here
v1.79.7
See changes here
v1.79.6
See changes here
v1.79.5
See changes here
v1.79.4
See changes here
v1.79.3
See changes here
v1.79.2
See changes here
v1.79.1
See changes here
v1.79.0
See changes here
v1.78.1
See changes here
v1.78.0
See changes here
v1.77.2
See changes here
v1.77.1
See changes here
v1.77.0
See changes here
v1.76.1
See changes here
v1.76.0
See changes here
v1.75.1
See changes here
v1.75.0
See changes here
v1.74.0
See changes here
v1.73.1
See changes here
v1.73.0
See changes here
v1.72.0
See changes here
v1.71.0
See changes here
v1.70.0
See changes here
v1.69.0
See changes here
v1.68.0
See changes here
v1.67.0
See changes here
v1.66.2
See changes here
v1.66.1
See changes here
v1.66.0
See changes here
v1.65.0
See changes here
v1.64.1
See changes here
v1.64.0
See changes here
v1.63.0
See changes here
v1.62.0
See changes here
v1.61.1
See changes here
v1.61.0
See changes here
v1.60.0
See changes here
v1.59.0
See changes here
v1.58.0
See changes here
v1.57.1
See changes here
v1.57.0
See changes here
v1.56.0
See changes here
v1.55.1
See changes here
v1.55.0
See changes here
v1.54.1
See changes here
v1.54.0
See changes here
v1.53.1
See changes here
v1.53.0
See changes here
v1.52.0
See changes here
v1.51.0
See changes here
v1.50.2
See changes here
v1.50.1
See changes here
v1.50.0
See changes here
v1.49.0
See changes here
v1.48.0
See changes here
v1.47.0
See changes here
v1.46.0
See changes here
v1.45.0
See changes here
v1.44.0
See changes here
v1.43.0
See changes here
v1.42.0
See changes here
Previous releases
See releases page.