mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 00:13:30 +01:00

Aliaksandr Valialkin c1a3876e95

docs/MetricsQL.md: properly document how MetricsQL selects the lookbehind window

- rate(m) isn't equivalent to rate(m[1i]) when step is smaller than the interval between samples.
- default_rollup(m) isn't equivalent to default_rollup(m[1i]) when step is smaller than the interval between samples.

These changes have been made in v1.85.3 as a part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483 .
See the corresponding commit - 9fa3f1dc57 .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5816

2024-02-18 13:48:45 +02:00

122 KiB

Raw Blame History

sort

weight

title

aliases

MetricsQL

docs

parent	weight
victoriametrics	23

/ExtendedPromQL.html

/MetricsQL.html

MetricsQL

VictoriaMetrics implements MetricsQL - query language inspired by PromQL. MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work the same after switching from Prometheus to VictoriaMetrics. However, there are some intentional differences between these two languages.

Standalone MetricsQL package can be used for parsing MetricsQL in external apps.

If you are unfamiliar with PromQL, then it is suggested reading this tutorial for beginners and introduction into basic querying via MetricsQL.

The following functionality is implemented differently in MetricsQL compared to PromQL. This improves user experience:

MetricsQL takes into account the last raw sample before the lookbehind window in square brackets for increase and rate functions. This allows returning the exact results users expect for increase(metric[$__interval]) queries instead of incomplete results Prometheus returns for such queries. Prometheus misses the increase between the last sample before the lookbehind window and the first sample inside the lookbehind window.
MetricsQL doesn't extrapolate rate and increase function results, so it always returns the expected results. For example, it returns integer results from increase() over slow-changing integer counter. Prometheus in this case returns unexpected fractional results, which may significantly differ from the expected results. This addresses this issue from Prometheus. See technical details about VictoriaMetrics and Prometheus calculations for rate and increase in this issue.
MetricsQL returns the expected non-empty responses for rate function when Grafana or vmui passes step values smaller than the interval between raw samples to /api/v1/query_range. This addresses this issue from Grafana. See also this blog post.
MetricsQL treats scalar type the same as instant vector without labels, since subtle differences between these types usually confuse users. See the corresponding Prometheus docs for details.
MetricsQL removes all the NaN values from the output, so some queries like (-1)^0.5 return empty results in VictoriaMetrics, while returning a series of NaN values in Prometheus. Note that Grafana doesn't draw any lines or dots for NaN values, so the end result looks the same for both VictoriaMetrics and Prometheus.
MetricsQL keeps metric names after applying functions, which don't change the meaning of the original time series. For example, min_over_time(foo) or round(foo) leaves foo metric name in the result. See this issue for details.

Read more about the differences between PromQL and MetricsQL in this article.

Other PromQL functionality should work the same in MetricsQL. File an issue if you notice discrepancies between PromQL and MetricsQL results other than mentioned above.

MetricsQL features

MetricsQL implements PromQL and provides additional functionality mentioned below, which is aimed towards solving practical cases. Feel free filing a feature request if you think MetricsQL misses certain useful functionality.

This functionality can be evaluated at VictoriaMetrics playground or at your own VictoriaMetrics instance.

The list of MetricsQL features on top of PromQL:

Graphite-compatible filters can be passed via {__graphite__="foo.*.bar"} syntax. See these docs. VictoriaMetrics can be used as Graphite datasource in Grafana. See these docs for details. See also label_graphite_group function, which can be used for extracting the given groups from Graphite metric name.
Lookbehind window in square brackets for rollup functions may be omitted. VictoriaMetrics automatically selects the lookbehind window depending on the step query arg passed to /api/v1/query_range and the real interval between raw samples (aka scrape_interval). For instance, the following query is valid in VictoriaMetrics: rate(node_network_receive_bytes_total). It is roughly equivalent to rate(node_network_receive_bytes_total[$__interval]) when used in Grafana. The difference is documented in rate() docs.
Numeric values can contain _ delimiters for better readability. For example, 1_234_567_890 can be used in queries instead of 1234567890.
Series selectors accept multiple or filters. For example, {env="prod",job="a" or env="dev",job="b"} selects series with {env="prod",job="a"} or {env="dev",job="b"} labels. See these docs for details.
Support for group_left(*) and group_right(*) for copying all the labels from time series on the one side of many-to-one operations. The copied label names may clash with the existing label names, so MetricsQL provides an ability to add prefix to the copied metric names via group_left(*) prefix "..." syntax. For example, the following query copies all the namespace-related labels from kube_namespace_labels to kube_pod_info series, while adding ns_ prefix to the copied labels: kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels. Labels from the on() list aren't copied.
Aggregate functions accept arbitrary number of args. For example, avg(q1, q2, q3) would return the average values for every point across time series returned by q1, q2 and q3.
@ modifier can be put anywhere in the query. For example, sum(foo) @ end() calculates sum(foo) at the end timestamp of the selected time range [start ... end].
Arbitrary subexpression can be used as @ modifier. For example, foo @ (end() - 1h) calculates foo at the end - 1 hour timestamp on the selected time range [start ... end].
offset, lookbehind window in square brackets and step value for subquery may refer to the current step aka $__interval value from Grafana with [Ni] syntax. For instance, rate(metric[10i] offset 5i) would return per-second rate over a range covering 10 previous steps with the offset of 5 steps.
offset may be put anywhere in the query. For instance, sum(foo) offset 24h.
Lookbehind window in square brackets and offset may be fractional. For instance, rate(node_network_receive_bytes_total[1.5m] offset 0.5d).
The duration suffix is optional. The duration is in seconds if the suffix is missing. For example, rate(m[300] offset 1800) is equivalent to rate(m[5m]) offset 30m.
The duration can be placed anywhere in the query. For example, sum_over_time(m[1h]) / 1h is equivalent to sum_over_time(m[1h]) / 3600.
Numeric values can have K, Ki, M, Mi, G, Gi, T and Ti suffixes. For example, 8K is equivalent to 8000, while 1.2Mi is equivalent to 1.2*1024*1024.
Trailing commas on all the lists are allowed - label filters, function args and with expressions. For instance, the following queries are valid: m{foo="bar",}, f(a, b,), WITH (x=y,) x. This simplifies maintenance of multi-line queries.
Metric names and label names may contain any unicode letter. For example температура{город="Киев"} is a value MetricsQL expression.
Metric names and labels names may contain escaped chars. For example, foo\-bar{baz\=aa="b"} is valid expression. It returns time series with name foo-bar containing label baz=aa with value b. Additionally, the following escape sequences are supported:
- \xXX, where XX is hexadecimal representation of the escaped ascii char.
- \uXXXX, where XXXX is a hexadecimal representation of the escaped unicode char.
Aggregate functions support optional limit N suffix in order to limit the number of output series. For example, sum(x) by (y) limit 3 limits the number of output time series after the aggregation to 3. All the other time series are dropped.
histogram_quantile accepts optional third arg - boundsLabel. In this case it returns lower and upper bounds for the estimated percentile. See this issue for details.
default binary operator. q1 default q2 fills gaps in q1 with the corresponding values from q2. See also drop_empty_series.
if binary operator. q1 if q2 removes values from q1 for missing values from q2.
ifnot binary operator. q1 ifnot q2 removes values from q1 for existing values from q2.
WITH templates. This feature simplifies writing and managing complex queries. Go to WITH templates playground and try it.
String literals may be concatenated. This is useful with WITH templates: WITH (commonPrefix="long_metric_prefix_") {__name__=commonPrefix+"suffix1"} / {__name__=commonPrefix+"suffix2"}.
keep_metric_names modifier can be applied to all the rollup functions, transform functions and binary operators. This modifier prevents from dropping metric names in function results. See these docs.

keep_metric_names

By default, metric names are dropped after applying functions or binary operators, since they may change the meaning of the original time series. This may result in duplicate time series error when the function is applied to multiple time series with different names. This error can be fixed by applying keep_metric_names modifier to the function or binary operator.

For example:

rate({__name__=~"foo|bar"}) keep_metric_names leaves foo and bar metric names in the returned time series.
({__name__=~"foo|bar"} / 10) keep_metric_names leaves foo and bar metric names in the returned time series.

MetricsQL functions

If you are unfamiliar with PromQL, then please read this tutorial at first.

MetricsQL provides the following functions:

Rollup functions
Transform functions
Label manipulation functions
Aggregate functions

Rollup functions

Rollup functions (aka range functions or window functions) calculate rollups over raw samples on the given lookbehind window for the selected time series. For example, avg_over_time(temperature[24h]) calculates the average temperature over raw samples for the last 24 hours.

Additional details:

If rollup functions are used for building graphs in Grafana, then the rollup is calculated independently per each point on the graph. For example, every point for avg_over_time(temperature[24h]) graph shows the average temperature for the last 24 hours ending at this point. The interval between points is set as step query arg passed by Grafana to /api/v1/query_range.
If the given series selector returns multiple time series, then rollups are calculated individually per each returned series.
If lookbehind window in square brackets is missing, then it is automatically set to the following value:
- To step value passed to /api/v1/query_range or /api/v1/query for all the rollup functions except of default_rollup and rate. This value is known as $__interval in Grafana or 1i in MetricsQL. For example, avg_over_time(temperature) is automatically transformed to avg_over_time(temperature[1i]).
- To the max(step, scrape_interval), where scrape_interval is the interval between raw samples for default_rollup and rate functions. This allows avoiding unexpected gaps on the graph when step is smaller than scrape_interval.
Every series selector in MetricsQL must be wrapped into a rollup function. Otherwise, it is automatically wrapped into default_rollup. For example, foo{bar="baz"} is automatically converted to default_rollup(foo{bar="baz"}) before performing the calculations.
If something other than series selector is passed to rollup function, then the inner arg is automatically converted to a subquery.
All the rollup functions accept optional keep_metric_names modifier. If it is set, then the function keeps metric names in results. See these docs.

See also implicit query conversions.

The list of supported rollup functions:

absent_over_time

absent_over_time(series_selector[d]) is a rollup function, which returns 1 if the given lookbehind window d doesn't contain raw samples. Otherwise, it returns an empty result.

This function is supported by PromQL. See also present_over_time.

aggr_over_time

aggr_over_time(("rollup_func1", "rollup_func2", ...), series_selector[d]) is a rollup function, which calculates all the listed rollup_func* for raw samples on the given lookbehind window d. The calculations are performed individually per each time series returned from the given series_selector.

rollup_func* can contain any rollup function. For instance, aggr_over_time(("min_over_time", "max_over_time", "rate"), m[d]) would calculate min_over_time, max_over_time and rate for m[d].

ascent_over_time

ascent_over_time(series_selector[d]) is a rollup function, which calculates ascent of raw sample values on the given lookbehind window d. The calculations are performed individually per each time series returned from the given series_selector.

This function is useful for tracking height gains in GPS tracking. Metric names are stripped from the resulting rollups.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

avg_over_time

avg_over_time(series_selector[d]) is a rollup function, which calculates the average value over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

This function is supported by PromQL. See also median_over_time.

changes

changes(series_selector[d]) is a rollup function, which calculates the number of times the raw samples changed on the given lookbehind window d per each time series returned from the given series_selector.

Unlike changes() in Prometheus it takes into account the change from the last sample before the given lookbehind window d. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also changes_prometheus.

changes_prometheus

changes_prometheus(series_selector[d]) is a rollup function, which calculates the number of times the raw samples changed on the given lookbehind window d per each time series returned from the given series_selector.

It doesn't take into account the change from the last sample before the given lookbehind window d in the same way as Prometheus does. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also changes.

count_eq_over_time

count_eq_over_time(series_selector[d], eq) is a rollup function, which calculates the number of raw samples on the given lookbehind window d, which are equal to eq. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also count_over_time and share_eq_over_time.

count_gt_over_time

count_gt_over_time(series_selector[d], gt) is a rollup function, which calculates the number of raw samples on the given lookbehind window d, which are bigger than gt. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also count_over_time and share_gt_over_time.

count_le_over_time

count_le_over_time(series_selector[d], le) is a rollup function, which calculates the number of raw samples on the given lookbehind window d, which don't exceed le. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also count_over_time and share_le_over_time.

count_ne_over_time

count_ne_over_time(series_selector[d], ne) is a rollup function, which calculates the number of raw samples on the given lookbehind window d, which aren't equal to ne. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

count_over_time

count_over_time(series_selector[d]) is a rollup function, which calculates the number of raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also count_le_over_time, count_gt_over_time, count_eq_over_time and count_ne_over_time.

decreases_over_time

decreases_over_time(series_selector[d]) is a rollup function, which calculates the number of raw sample value decreases over the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

default_rollup

default_rollup(series_selector[d]) is a rollup function, which returns the last raw sample value on the given lookbehind window d per each time series returned from the given series_selector.

If the lookbehind window is skipped in square brackets, then it is automatically calculated as max(step, scrape_interval), where step is the query arg value passed to /api/v1/query_range or /api/v1/query, while scrape_interval is the interval between raw samples for the selected time series. This allows avoiding unexpected gaps on the graph when step is smaller than the scrape_interval.

delta

delta(series_selector[d]) is a rollup function, which calculates the difference between the last sample before the given lookbehind window d and the last sample at the given lookbehind window d per each time series returned from the given series_selector.

The behaviour of delta() function in MetricsQL is slightly different to the behaviour of delta() function in Prometheus. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also increase and delta_prometheus.

delta_prometheus

delta_prometheus(series_selector[d]) is a rollup function, which calculates the difference between the first and the last samples at the given lookbehind window d per each time series returned from the given series_selector.

The behaviour of delta_prometheus() is close to the behaviour of delta() function in Prometheus. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

deriv

deriv(series_selector[d]) is a rollup function, which calculates per-second derivative over the given lookbehind window d per each time series returned from the given series_selector. The derivative is calculated using linear regression.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also deriv_fast and ideriv.

deriv_fast

deriv_fast(series_selector[d]) is a rollup function, which calculates per-second derivative using the first and the last raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

descent_over_time

descent_over_time(series_selector[d]) is a rollup function, which calculates descent of raw sample values on the given lookbehind window d. The calculations are performed individually per each time series returned from the given series_selector.

This function is useful for tracking height loss in GPS tracking.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

distinct_over_time

distinct_over_time(series_selector[d]) is a rollup function, which returns the number of distinct raw sample values on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

duration_over_time

duration_over_time(series_selector[d], max_interval) is a rollup function, which returns the duration in seconds when time series returned from the given series_selector were present over the given lookbehind window d. It is expected that intervals between adjacent samples per each series don't exceed the max_interval. Otherwise, such intervals are considered as gaps and aren't counted.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

first_over_time

first_over_time(series_selector[d]) is a rollup function, which returns the first raw sample value on the given lookbehind window d per each time series returned from the given series_selector.

See also last_over_time and tfirst_over_time.

geomean_over_time

geomean_over_time(series_selector[d]) is a rollup function, which calculates geometric mean over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

histogram_over_time

histogram_over_time(series_selector[d]) is a rollup function, which calculates VictoriaMetrics histogram over raw samples on the given lookbehind window d. It is calculated individually per each time series returned from the given series_selector. The resulting histograms are useful to pass to histogram_quantile for calculating quantiles over multiple gauges. For example, the following query calculates median temperature by country over the last 24 hours:

histogram_quantile(0.5, sum(histogram_over_time(temperature[24h])) by (vmrange,country)).

hoeffding_bound_lower

hoeffding_bound_lower(phi, series_selector[d]) is a rollup function, which calculates lower Hoeffding bound for the given phi in the range [0...1].

hoeffding_bound_upper

hoeffding_bound_upper(phi, series_selector[d]) is a rollup function, which calculates upper Hoeffding bound for the given phi in the range [0...1].

holt_winters

holt_winters(series_selector[d], sf, tf) is a rollup function, which calculates Holt-Winters value (aka double exponential smoothing) for raw samples over the given lookbehind window d using the given smoothing factor sf and the given trend factor tf. Both sf and tf must be in the range [0...1]. It is expected that the series_selector returns time series of gauge type.

This function is supported by PromQL. See also range_linear_regression.

idelta

idelta(series_selector[d]) is a rollup function, which calculates the difference between the last two raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also delta.

ideriv

ideriv(series_selector[d]) is a rollup function, which calculates the per-second derivative based on the last two raw samples over the given lookbehind window d. The derivative is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

increase

increase(series_selector[d]) is a rollup function, which calculates the increase over the given lookbehind window d per each time series returned from the given series_selector. It is expected that the series_selector returns time series of counter type.

Unlike Prometheus, it takes into account the last sample before the given lookbehind window d when calculating the result. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also increase_pure, increase_prometheus and delta.

increase_prometheus

increase_prometheus(series_selector[d]) is a rollup function, which calculates the increase over the given lookbehind window d per each time series returned from the given series_selector. It is expected that the series_selector returns time series of counter type. It doesn't take into account the last sample before the given lookbehind window d when calculating the result in the same way as Prometheus does. See this article for details.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

increase_pure

increase_pure(series_selector[d]) is a rollup function, which works the same as increase except of the following corner case - it assumes that counters always start from 0, while increase ignores the first value in a series if it is too big.

increases_over_time

increases_over_time(series_selector[d]) is a rollup function, which calculates the number of raw sample value increases over the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

integrate

integrate(series_selector[d]) is a rollup function, which calculates the integral over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

irate

irate(series_selector[d]) is a rollup function, which calculates the "instant" per-second increase rate over the last two raw samples on the given lookbehind window d per each time series returned from the given series_selector. It is expected that the series_selector returns time series of counter type.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also rate and rollup_rate.

lag

lag(series_selector[d]) is a rollup function, which returns the duration in seconds between the last sample on the given lookbehind window d and the timestamp of the current point. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also lifetime and duration_over_time.

last_over_time

last_over_time(series_selector[d]) is a rollup function, which returns the last raw sample value on the given lookbehind window d per each time series returned from the given series_selector.

This function is supported by PromQL. See also first_over_time and tlast_over_time.

lifetime

lifetime(series_selector[d]) is a rollup function, which returns the duration in seconds between the last and the first sample on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

mad_over_time

mad_over_time(series_selector[d]) is a rollup function, which calculates median absolute deviation over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

See also mad, range_mad and outlier_iqr_over_time.

max_over_time

max_over_time(series_selector[d]) is a rollup function, which calculates the maximum value over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

This function is supported by PromQL. See also tmax_over_time.

median_over_time

median_over_time(series_selector[d]) is a rollup function, which calculates median value over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

min_over_time

min_over_time(series_selector[d]) is a rollup function, which calculates the minimum value over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

This function is supported by PromQL. See also tmin_over_time.

mode_over_time

mode_over_time(series_selector[d]) is a rollup function, which calculates mode for raw samples on the given lookbehind window d. It is calculated individually per each time series returned from the given series_selector. It is expected that raw sample values are discrete.

outlier_iqr_over_time

outlier_iqr_over_time(series_selector[d]) is a rollup function, which returns the last sample on the given lookbehind window d if its value is either smaller than the q25-1.5*iqr or bigger than q75+1.5*iqr where:

iqr is an Interquartile range over raw samples on the lookbehind window d
q25 and q75 are 25th and 75th percentiles over raw samples on the lookbehind window d.

The outlier_iqr_over_time() is useful for detecting anomalies in gauge values based on the previous history of values. For example, outlier_iqr_over_time(memory_usage_bytes[1h]) triggers when memory_usage_bytes suddenly goes outside the usual value range for the last 24 hours.

predict_linear

predict_linear(series_selector[d], t) is a rollup function, which calculates the value t seconds in the future using linear interpolation over raw samples on the given lookbehind window d. The predicted value is calculated individually per each time series returned from the given series_selector.

This function is supported by PromQL. See also range_linear_regression.

present_over_time

present_over_time(series_selector[d]) is a rollup function, which returns 1 if there is at least a single raw sample on the given lookbehind window d. Otherwise, an empty result is returned.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

quantile_over_time

quantile_over_time(phi, series_selector[d]) is a rollup function, which calculates phi-quantile over raw samples on the given lookbehind window d per each time series returned from the given series_selector. The phi value must be in the range [0...1].

This function is supported by PromQL. See also quantiles_over_time.

quantiles_over_time

quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d]) is a rollup function, which calculates phi*-quantiles over raw samples on the given lookbehind window d per each time series returned from the given series_selector. The function returns individual series per each phi* with {phiLabel="phi*"} label. phi* values must be in the range [0...1].

range_over_time

range_over_time(series_selector[d]) is a rollup function, which calculates value range over raw samples on the given lookbehind window d per each time series returned from the given series_selector. E.g. it calculates max_over_time(series_selector[d]) - min_over_time(series_selector[d]).

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

rate

rate(series_selector[d]) is a rollup function, which calculates the average per-second increase rate over the given lookbehind window d per each time series returned from the given series_selector. It is expected that the series_selector returns time series of counter type.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also irate and rollup_rate.

rate_over_sum

rate_over_sum(series_selector[d]) is a rollup function, which calculates per-second rate over the sum of raw samples on the given lookbehind window d. The calculations are performed individually per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

resets

resets(series_selector[d]) is a rollup function, which returns the number of counter resets over the given lookbehind window d per each time series returned from the given series_selector. It is expected that the series_selector returns time series of counter type.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

rollup

rollup(series_selector[d]) is a rollup function, which calculates min, max and avg values for raw samples on the given lookbehind window d and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels. These values are calculated individually per each time series returned from the given series_selector.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

rollup_candlestick

rollup_candlestick(series_selector[d]) is a rollup function, which calculates open, high, low and close values (aka OHLC) over raw samples on the given lookbehind window d and returns them in time series with rollup="open", rollup="high", rollup="low" and rollup="close" additional labels. The calculations are performed individually per each time series returned from the given series_selector. This function is useful for financial applications.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

rollup_delta

rollup_delta(series_selector[d]) is a rollup function, which calculates differences between adjacent raw samples on the given lookbehind window d and returns min, max and avg values for the calculated differences and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels. The calculations are performed individually per each time series returned from the given series_selector.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

rollup_deriv

rollup_deriv(series_selector[d]) is a rollup function, which calculates per-second derivatives for adjacent raw samples on the given lookbehind window d and returns min, max and avg values for the calculated per-second derivatives and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels. The calculations are performed individually per each time series returned from the given series_selector.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

rollup_increase

rollup_increase(series_selector[d]) is a rollup function, which calculates increases for adjacent raw samples on the given lookbehind window d and returns min, max and avg values for the calculated increases and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels. The calculations are performed individually per each time series returned from the given series_selector.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names. See also rollup_delta.

rollup_rate

rollup_rate(series_selector[d]) is a rollup function, which calculates per-second change rates for adjacent raw samples on the given lookbehind window d and returns min, max and avg values for the calculated per-second change rates and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels.

See this article in order to understand better when to use rollup_rate().

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

The calculations are performed individually per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

rollup_scrape_interval

rollup_scrape_interval(series_selector[d]) is a rollup function, which calculates the interval in seconds between adjacent raw samples on the given lookbehind window d and returns min, max and avg values for the calculated interval and returns them in time series with rollup="min", rollup="max" and rollup="avg" additional labels. The calculations are performed individually per each time series returned from the given series_selector.

Optional 2nd argument "min", "max" or "avg" can be passed to keep only one calculation result and without adding a label.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names. See also scrape_interval.

scrape_interval

scrape_interval(series_selector[d]) is a rollup function, which calculates the average interval in seconds between raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

share_gt_over_time

share_gt_over_time(series_selector[d], gt) is a rollup function, which returns share (in the range [0...1]) of raw samples on the given lookbehind window d, which are bigger than gt. It is calculated independently per each time series returned from the given series_selector.

This function is useful for calculating SLI and SLO. Example: share_gt_over_time(up[24h], 0) - returns service availability for the last 24 hours.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

share_le_over_time

share_le_over_time(series_selector[d], le) is a rollup function, which returns share (in the range [0...1]) of raw samples on the given lookbehind window d, which are smaller or equal to le. It is calculated independently per each time series returned from the given series_selector.

This function is useful for calculating SLI and SLO. Example: share_le_over_time(memory_usage_bytes[24h], 100*1024*1024) returns the share of time series values for the last 24 hours when memory usage was below or equal to 100MB.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

share_eq_over_time

share_eq_over_time(series_selector[d], eq) is a rollup function, which returns share (in the range [0...1]) of raw samples on the given lookbehind window d, which are equal to eq. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

stale_samples_over_time

stale_samples_over_time(series_selector[d]) is a rollup function, which calculates the number of staleness markers on the given lookbehind window d per each time series matching the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

stddev_over_time

stddev_over_time(series_selector[d]) is a rollup function, which calculates standard deviation over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also stdvar_over_time.

stdvar_over_time

stdvar_over_time(series_selector[d]) is a rollup function, which calculates standard variance over raw samples on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also stddev_over_time.

sum_eq_over_time

sum_eq_over_time(series_selector[d], eq) is a rollup function, which calculates the sum of raw sample values equal to eq on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also sum_over_time and count_eq_over_time.

sum_gt_over_time

sum_gt_over_time(series_selector[d], gt) is a rollup function, which calculates the sum of raw sample values bigger than gt on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also sum_over_time and count_gt_over_time.

sum_le_over_time

sum_le_over_time(series_selector[d], le) is a rollup function, which calculates the sum of raw sample values smaller or equal to le on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

See also sum_over_time and count_le_over_time.

sum_over_time

sum_over_time(series_selector[d]) is a rollup function, which calculates the sum of raw sample values on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

sum2_over_time

sum2_over_time(series_selector[d]) is a rollup function, which calculates the sum of squares for raw sample values on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

timestamp

timestamp(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the last raw sample on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also timestamp_with_name.

timestamp_with_name

timestamp_with_name(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the last raw sample on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are preserved in the resulting rollups.

tfirst_over_time

tfirst_over_time(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the first raw sample on the given lookbehind window d per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

tlast_change_over_time

tlast_change_over_time(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the last change per each time series returned from the given series_selector on the given lookbehind window d.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

tlast_over_time

tlast_over_time(series_selector[d]) is a rollup function, which is an alias for timestamp.

tmax_over_time

tmax_over_time(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the raw sample with the maximum value on the given lookbehind window d. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

tmin_over_time

tmin_over_time(series_selector[d]) is a rollup function, which returns the timestamp in seconds with millisecond precision for the raw sample with the minimum value on the given lookbehind window d. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

zscore_over_time

zscore_over_time(series_selector[d]) is a rollup function, which returns z-score for raw samples on the given lookbehind window d. It is calculated independently per each time series returned from the given series_selector.

Metric names are stripped from the resulting rollups. Add keep_metric_names modifier in order to keep metric names.

Transform functions

Transform functions calculate transformations over rollup results. For example, abs(delta(temperature[24h])) calculates the absolute value for every point of every time series returned from the rollup delta(temperature[24h]).

Additional details:

If transform function is applied directly to a series selector, then the default_rollup() function is automatically applied before calculating the transformations. For example, abs(temperature) is implicitly transformed to abs(default_rollup(temperature)).
All the transform functions accept optional keep_metric_names modifier. If it is set, then the function doesn't drop metric names from the resulting time series. See these docs.

See also implicit query conversions.

The list of supported transform functions:

abs

abs(q) is a transform function, which calculates the absolute value for every point of every time series returned by q.

This function is supported by PromQL.

absent

absent(q) is a transform function, which returns 1 if q has no points. Otherwise, returns an empty result.

This function is supported by PromQL. See also absent_over_time.

acos

acos(q) is a transform function, which returns inverse cosine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also asin and cos.

acosh

acosh(q) is a transform function, which returns inverse hyperbolic cosine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also sinh.

asin

asin(q) is a transform function, which returns inverse sine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also acos and sin.

asinh

asinh(q) is a transform function, which returns inverse hyperbolic sine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also sinh.

atan

atan(q) is a transform function, which returns inverse tangent for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also tan.

atanh

atanh(q) is a transform function, which returns inverse hyperbolic tangent for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also tanh.

bitmap_and

bitmap_and(q, mask) is a transform function, which calculates bitwise v & mask for every v point of every time series returned from q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

bitmap_or

bitmap_or(q, mask) is a transform function, which calculates bitwise v | mask for every v point of every time series returned from q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

bitmap_xor

bitmap_xor(q, mask) is a transform function, which calculates bitwise v ^ mask for every v point of every time series returned from q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

buckets_limit

buckets_limit(limit, buckets) is a transform function, which limits the number of histogram buckets to the given limit.

ceil

ceil(q) is a transform function, which rounds every point for every time series returned by q to the upper nearest integer.

This function is supported by PromQL. See also floor and round.

clamp

clamp(q, min, max) is a transform function, which clamps every point for every time series returned by q with the given min and max values.

This function is supported by PromQL. See also clamp_min and clamp_max.

clamp_max

clamp_max(q, max) is a transform function, which clamps every point for every time series returned by q with the given max value.

This function is supported by PromQL. See also clamp and clamp_min.

clamp_min

clamp_min(q, min) is a transform function, which clamps every point for every time series returned by q with the given min value.

This function is supported by PromQL. See also clamp and clamp_max.

cos

cos(q) is a transform function, which returns cos(v) for every v point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also sin.

cosh

cosh(q) is a transform function, which returns hyperbolic cosine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also acosh.

day_of_month

day_of_month(q) is a transform function, which returns the day of month for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [1...31].

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

day_of_week

day_of_week(q) is a transform function, which returns the day of week for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [0...6], where 0 means Sunday and 6 means Saturday.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

day_of_year

day_of_year(q) is a transform function, which returns the day of year for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [1...365] for non-leap years, and [1 to 366] in leap years.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

days_in_month

days_in_month(q) is a transform function, which returns the number of days in the month identified by every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [28...31].

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

deg

deg(q) is a transform function, which converts Radians to degrees for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also rad.

drop_empty_series

drop_empty_series(q) is a transform function, which drops empty series from q.

This function can be used when default operator should be applied only to non-empty series. For example, drop_empty_series(temperature < 30) default 42 returns series, which have at least a single sample smaller than 30 on the selected time range, while filling gaps in the returned series with 42.

On the other hand (temperature < 30) default 40 returns all the temperature series, even if they have no samples smaller than 30, by replacing all the values bigger or equal to 30 with 40.

end

end() is a transform function, which returns the unix timestamp in seconds for the last point. It is known as end query arg passed to /api/v1/query_range.

exp

exp(q) is a transform function, which calculates the e^v for every point v of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also ln.

floor

floor(q) is a transform function, which rounds every point for every time series returned by q to the lower nearest integer.

This function is supported by PromQL. See also ceil and round.

histogram_avg

histogram_avg(buckets) is a transform function, which calculates the average value for the given buckets. It can be used for calculating the average over the given time range across multiple time series. For example, histogram_avg(sum(histogram_over_time(response_time_duration_seconds[5m])) by (vmrange,job)) would return the average response time per each job over the last 5 minutes.

histogram_quantile

histogram_quantile(phi, buckets) is a transform function, which calculates phi-percentile over the given histogram buckets. phi must be in the range [0...1]. For example, histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) would return median request duration for all the requests during the last 5 minutes.

The function accepts optional third arg - boundsLabel. In this case it returns lower and upper bounds for the estimated percentile with the given boundsLabel label. See this issue for details.

When the percentile is calculated over multiple histograms, then all the input histograms must have buckets with identical boundaries, e.g. they must have the same set of le or vmrange labels. Otherwise, the returned result may be invalid. See this issue for details.

This function is supported by PromQL (except of the boundLabel arg). See also histogram_quantiles, histogram_share and quantile.

histogram_quantiles

histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) is a transform function, which calculates the given phi*-quantiles over the given histogram buckets. Argument phi* must be in the range [0...1]. For example, histogram_quantiles('le', 0.3, 0.5, sum(rate(http_request_duration_seconds_bucket[5m]) by (le)). Each calculated quantile is returned in a separate time series with the corresponding {phiLabel="phi*"} label.

histogram_share

histogram_share(le, buckets) is a transform function, which calculates the share (in the range [0...1]) for buckets that fall below le. This function is useful for calculating SLI and SLO. This is inverse to histogram_quantile.

The function accepts optional third arg - boundsLabel. In this case it returns lower and upper bounds for the estimated share with the given boundsLabel label.

histogram_stddev

histogram_stddev(buckets) is a transform function, which calculates standard deviation for the given buckets.

histogram_stdvar

histogram_stdvar(buckets) is a transform function, which calculates standard variance for the given buckets. It can be used for calculating standard deviation over the given time range across multiple time series. For example, histogram_stdvar(sum(histogram_over_time(temperature[24])) by (vmrange,country)) would return standard deviation for the temperature per each country over the last 24 hours.

hour

hour(q) is a transform function, which returns the hour for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [0...23].

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

interpolate

interpolate(q) is a transform function, which fills gaps with linearly interpolated values calculated from the last and the next non-empty points per each time series returned by q.

See also keep_last_value and keep_next_value.

keep_last_value

keep_last_value(q) is a transform function, which fills gaps with the value of the last non-empty point in every time series returned by q.

See also keep_next_value and interpolate.

keep_next_value

keep_next_value(q) is a transform function, which fills gaps with the value of the next non-empty point in every time series returned by q.

See also keep_last_value and interpolate.

limit_offset

limit_offset(limit, offset, q) is a transform function, which skips offset time series from series returned by q and then returns up to limit of the remaining time series per each group.

This allows implementing simple paging for q time series. See also limitk.

ln

ln(q) is a transform function, which calculates ln(v) for every point v of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also exp and log2.

log2

log2(q) is a transform function, which calculates log2(v) for every point v of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also log10 and ln.

log10

log10(q) is a transform function, which calculates log10(v) for every point v of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also log2 and ln.

minute

minute(q) is a transform function, which returns the minute for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [0...59].

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

month

month(q) is a transform function, which returns the month for every point of every time series returned by q. It is expected that q returns unix timestamps. The returned values are in the range [1...12], where 1 means January and 12 means December.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

now

now() is a transform function, which returns the current timestamp as a floating-point value in seconds.

pi

pi() is a transform function, which returns Pi number.

This function is supported by PromQL.

rad

rad(q) is a transform function, which converts degrees to Radians for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL. See also deg.

prometheus_buckets

prometheus_buckets(buckets) is a transform function, which converts VictoriaMetrics histogram buckets with vmrange labels to Prometheus histogram buckets with le labels. This may be useful for building heatmaps in Grafana.

See also histogram_quantile and buckets_limit.

rand

rand(seed) is a transform function, which returns pseudo-random numbers on the range [0...1] with even distribution. Optional seed can be used as a seed for pseudo-random number generator.

See also rand_normal and rand_exponential.

rand_exponential

rand_exponential(seed) is a transform function, which returns pseudo-random numbers with exponential distribution. Optional seed can be used as a seed for pseudo-random number generator.

rand_normal

rand_normal(seed) is a transform function, which returns pseudo-random numbers with normal distribution. Optional seed can be used as a seed for pseudo-random number generator.

range_avg

range_avg(q) is a transform function, which calculates the avg value across points per each time series returned by q.

range_first

range_first(q) is a transform function, which returns the value for the first point per each time series returned by q.

range_last

range_last(q) is a transform function, which returns the value for the last point per each time series returned by q.

range_linear_regression

range_linear_regression(q) is a transform function, which calculates simple linear regression over the selected time range per each time series returned by q. This function is useful for capacity planning and predictions.

range_mad

range_mad(q) is a transform function, which calculates the median absolute deviation across points per each time series returned by q.

range_max

range_max(q) is a transform function, which calculates the max value across points per each time series returned by q.

range_median

range_median(q) is a transform function, which calculates the median value across points per each time series returned by q.

range_min

range_min(q) is a transform function, which calculates the min value across points per each time series returned by q.

range_normalize

range_normalize(q1, ...) is a transform function, which normalizes values for time series returned by q1, ... into [0 ... 1] range. This function is useful for correlating time series with distinct value ranges.

range_quantile

range_quantile(phi, q) is a transform function, which returns phi-quantile across points per each time series returned by q. phi must be in the range [0...1].

range_stddev

range_stddev(q) is a transform function, which calculates standard deviation per each time series returned by q on the selected time range.

range_stdvar

range_stdvar(q) is a transform function, which calculates standard variance per each time series returned by q on the selected time range.

range_sum

range_sum(q) is a transform function, which calculates the sum of points per each time series returned by q.

range_trim_outliers

range_trim_outliers(k, q) is a transform function, which drops points located farther than k*range_mad(q) from the range_median(q). E.g. it is equivalent to the following query: q ifnot (abs(q - range_median(q)) > k*range_mad(q)).

See also range_trim_spikes and range_trim_zscore.

range_trim_spikes

range_trim_spikes(phi, q) is a transform function, which drops phi percent of biggest spikes from time series returned by q. The phi must be in the range [0..1], where 0 means 0% and 1 means 100%.

range_trim_zscore

range_trim_zscore(z, q) is a transform function, which drops points located farther than z*range_stddev(q) from the range_avg(q). E.g. it is equivalent to the following query: q ifnot (abs(q - range_avg(q)) > z*range_avg(q)).

range_zscore

range_zscore(q) is a transform function, which calculates z-score for points returned by q, e.g. it is equivalent to the following query: (q - range_avg(q)) / range_stddev(q).

remove_resets

remove_resets(q) is a transform function, which removes counter resets from time series returned by q.

round

round(q, nearest) is a transform function, which rounds every point of every time series returned by q to the nearest multiple. If nearest is missing then the rounding is performed to the nearest integer.

This function is supported by PromQL. See also floor and ceil.

ru

ru(free, max) is a transform function, which calculates resource utilization in the range [0%...100%] for the given free and max resources. For instance, ru(node_memory_MemFree_bytes, node_memory_MemTotal_bytes) returns memory utilization over node_exporter metrics.

running_avg

running_avg(q) is a transform function, which calculates the running avg per each time series returned by q.

running_max

running_max(q) is a transform function, which calculates the running max per each time series returned by q.

running_min

running_min(q) is a transform function, which calculates the running min per each time series returned by q.

running_sum

running_sum(q) is a transform function, which calculates the running sum per each time series returned by q.

scalar

scalar(q) is a transform function, which returns q if q contains only a single time series. Otherwise, it returns nothing.

This function is supported by PromQL.

sgn

sgn(q) is a transform function, which returns 1 if v>0, -1 if v<0 and 0 if v==0 for every point v of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

sin

sin(q) is a transform function, which returns sin(v) for every v point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by MetricsQL. See also cos.

sinh

sinh(q) is a transform function, which returns hyperbolic sine for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by MetricsQL. See also cosh.

tan

tan(q) is a transform function, which returns tan(v) for every v point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by MetricsQL. See also atan.

tanh

tanh(q) is a transform function, which returns hyperbolic tangent for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by MetricsQL. See also atanh.

smooth_exponential

smooth_exponential(q, sf) is a transform function, which smooths points per each time series returned by q using exponential moving average with the given smooth factor sf.

sort

sort(q) is a transform function, which sorts series in ascending order by the last point in every time series returned by q.

This function is supported by PromQL. See also sort_desc and sort_by_label.

sort_desc

sort_desc(q) is a transform function, which sorts series in descending order by the last point in every time series returned by q.

This function is supported by PromQL. See also sort and sort_by_label.

sqrt

sqrt(q) is a transform function, which calculates square root for every point of every time series returned by q.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

start

start() is a transform function, which returns unix timestamp in seconds for the first point.

It is known as start query arg passed to /api/v1/query_range.

step

step() is a transform function, which returns the step in seconds (aka interval) between the returned points. It is known as step query arg passed to /api/v1/query_range.

time

time() is a transform function, which returns unix timestamp for every returned point.

This function is supported by PromQL. See also now, start and end.

timezone_offset

timezone_offset(tz) is a transform function, which returns offset in seconds for the given timezone tz relative to UTC. This can be useful when combining with datetime-related functions. For example, day_of_week(time()+timezone_offset("America/Los_Angeles")) would return weekdays for America/Los_Angeles time zone.

Special Local time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs.

See the list of supported timezones.

ttf

ttf(free) is a transform function, which estimates the time in seconds needed to exhaust free resources. For instance, ttf(node_filesystem_avail_byte) returns the time to storage space exhaustion. This function may be useful for capacity planning.

union

union(q1, ..., qN) is a transform function, which returns a union of time series returned from q1, ..., qN. The union function name can be skipped - the following queries are equivalent: union(q1, q2) and (q1, q2).

It is expected that each q* query returns time series with unique sets of labels. Otherwise, only the first time series out of series with identical set of labels is returned. Use alias and label_set functions for giving unique labelsets per each q* query:

vector

vector(q) is a transform function, which returns q, e.g. it does nothing in MetricsQL.

This function is supported by PromQL.

year

year(q) is a transform function, which returns the year for every point of every time series returned by q. It is expected that q returns unix timestamps.

Metric names are stripped from the resulting series. Add keep_metric_names modifier in order to keep metric names.

This function is supported by PromQL.

Label manipulation functions

Label manipulation functions perform manipulations with labels on the selected rollup results.

Additional details:

If label manipulation function is applied directly to a series_selector, then the default_rollup() function is automatically applied before performing the label transformation. For example, alias(temperature, "foo") is implicitly transformed to alias(default_rollup(temperature), "foo").

See also implicit query conversions.

The list of supported label manipulation functions:

alias

alias(q, "name") is label manipulation function, which sets the given name to all the time series returned by q. For example, alias(up, "foobar") would rename up series to foobar series.

drop_common_labels

drop_common_labels(q1, ...., qN) is label manipulation function, which drops common label="value" pairs among time series returned from q1, ..., qN.

label_copy

label_copy(q, "src_label1", "dst_label1", ..., "src_labelN", "dst_labelN") is label manipulation function, which copies label values from src_label* to dst_label* for all the time series returned by q. If src_label is empty, then the corresponding dst_label is left untouched.

label_del

label_del(q, "label1", ..., "labelN") is label manipulation function, which deletes the given label* labels from all the time series returned by q.

label_graphite_group

label_graphite_group(q, groupNum1, ... groupNumN) is label manipulation function, which replaces metric names returned from q with the given Graphite group values concatenated via . char.

For example, label_graphite_group({__graphite__="foo*.bar.*"}, 0, 2) would substitute foo<any_value>.bar.<other_value> metric names with foo<any_value>.<other_value>.

This function is useful for aggregating Graphite metrics with aggregate functions. For example, the following query would return per-app memory usage:

sum by (__name__) (
    label_graphite_group({__graphite__="app*.host*.memory_usage"}, 0)
)

label_join

label_join(q, "dst_label", "separator", "src_label1", ..., "src_labelN") is label manipulation function, which joins src_label* values with the given separator and stores the result in dst_label. This is performed individually per each time series returned by q. For example, label_join(up{instance="xxx",job="yyy"}, "foo", "-", "instance", "job") would store xxx-yyy label value into foo label.

This function is supported by PromQL.

label_keep

label_keep(q, "label1", ..., "labelN") is label manipulation function, which deletes all the labels except of the listed label* labels in all the time series returned by q.

label_lowercase

label_lowercase(q, "label1", ..., "labelN") is label manipulation function, which lowercases values for the given label* labels in all the time series returned by q.

label_map

label_map(q, "label", "src_value1", "dst_value1", ..., "src_valueN", "dst_valueN") is label manipulation function, which maps label values from src_* to dst* for all the time series returned by q.

label_match

label_match(q, "label", "regexp") is label manipulation function, which drops time series from q with label not matching the given regexp. This function can be useful after rollup-like functions, which may return multiple time series for every input series.

See also label_mismatch and labels_equal.

label_mismatch

label_mismatch(q, "label", "regexp") is label manipulation function, which drops time series from q with label matching the given regexp. This function can be useful after rollup-like functions, which may return multiple time series for every input series.

label_move

label_move(q, "src_label1", "dst_label1", ..., "src_labelN", "dst_labelN") is label manipulation function, which moves label values from src_label* to dst_label* for all the time series returned by q. If src_label is empty, then the corresponding dst_label is left untouched.

label_replace

label_replace(q, "dst_label", "replacement", "src_label", "regex") is label manipulation function, which applies the given regex to src_label and stores the replacement in dst_label if the given regex matches src_label. The replacement may contain references to regex captures such as $1, $2, etc. These references are substituted by the corresponding regex captures. For example, label_replace(up{job="node-exporter"}, "foo", "bar-$1", "job", "node-(.+)") would store bar-exporter label value into foo label.

This function is supported by PromQL.

label_set

label_set(q, "label1", "value1", ..., "labelN", "valueN") is label manipulation function, which sets {label1="value1", ..., labelN="valueN"} labels to all the time series returned by q.

label_transform

label_transform(q, "label", "regexp", "replacement") is label manipulation function, which substitutes all the regexp occurrences by the given replacement in the given label.

label_uppercase

label_uppercase(q, "label1", ..., "labelN") is label manipulation function, which uppercases values for the given label* labels in all the time series returned by q.

label_value

label_value(q, "label") is label manipulation function, which returns numeric values for the given label for every time series returned by q.

For example, if label_value(foo, "bar") is applied to foo{bar="1.234"}, then it will return a time series foo{bar="1.234"} with 1.234 value. Function will return no data for non-numeric label values.

labels_equal

labels_equal(q, "label1", "label2", ...) is label manipulation function, which returns q series with identical values for the listed labels "label1", "label2", etc.

See also label_match and label_mismatch.

sort_by_label

sort_by_label(q, "label1", ... "labelN") is label manipulation function, which sorts series in ascending order by the given set of labels. For example, sort_by_label(foo, "bar") would sort foo series by values of the label bar in these series.

sort_by_label_desc

sort_by_label_desc(q, "label1", ... "labelN") is label manipulation function, which sorts series in descending order by the given set of labels. For example, sort_by_label(foo, "bar") would sort foo series by values of the label bar in these series.

sort_by_label_numeric

sort_by_label_numeric(q, "label1", ... "labelN") is label manipulation function, which sorts series in ascending order by the given set of labels using numeric sort. For example, if foo series have bar label with values 1, 101, 15 and 2, then sort_by_label_numeric(foo, "bar") would return series in the following order of bar label values: 1, 2, 15 and 101.

sort_by_label_numeric_desc

sort_by_label_numeric_desc(q, "label1", ... "labelN") is label manipulation function, which sorts series in descending order by the given set of labels using numeric sort. For example, if foo series have bar label with values 1, 101, 15 and 2, then sort_by_label_numeric(foo, "bar") would return series in the following order of bar label values: 101, 15, 2 and 1.

Aggregate functions

Aggregate functions calculate aggregates over groups of rollup results.

Additional details:

By default, a single group is used for aggregation. Multiple independent groups can be set up by specifying grouping labels in by and without modifiers. For example, count(up) by (job) would group rollup results by job label value and calculate the count aggregate function independently per each group, while count(up) without (instance) would group rollup results by all the labels except instance before calculating count aggregate function independently per each group. Multiple labels can be put in by and without modifiers.
If the aggregate function is applied directly to a series_selector, then the default_rollup() function is automatically applied before calculating the aggregate. For example, count(up) is implicitly transformed to count(default_rollup(up)).
Aggregate functions accept arbitrary number of args. For example, avg(q1, q2, q3) would return the average values for every point across time series returned by q1, q2 and q3.
Aggregate functions support optional limit N suffix, which can be used for limiting the number of output groups. For example, sum(x) by (y) limit 3 limits the number of groups for the aggregation to 3. All the other groups are ignored.

See also implicit query conversions.

The list of supported aggregate functions:

any

any(q) by (group_labels) is aggregate function, which returns a single series per group_labels out of time series returned by q.

avg

avg(q) by (group_labels) is aggregate function, which returns the average value per group_labels for time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

bottomk

bottomk(k, q) is aggregate function, which returns up to k points with the smallest values across all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL. See also topk.

bottomk_avg

bottomk_avg(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the smallest averages. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, bottomk_avg(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the smallest averages plus a time series with {job="other"} label with the sum of the remaining series if any.

bottomk_last

bottomk_last(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the smallest last values. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, bottomk_max(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the smallest maximums plus a time series with {job="other"} label with the sum of the remaining series if any.

bottomk_max

bottomk_max(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the smallest maximums. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, bottomk_max(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the smallest maximums plus a time series with {job="other"} label with the sum of the remaining series if any.

bottomk_median

bottomk_median(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the smallest medians. If an optionalother_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, bottomk_median(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the smallest medians plus a time series with {job="other"} label with the sum of the remaining series if any.

bottomk_min

bottomk_min(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the smallest minimums. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, bottomk_min(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the smallest minimums plus a time series with {job="other"} label with the sum of the remaining series if any.

count

count(q) by (group_labels) is aggregate function, which returns the number of non-empty points per group_labels for time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

count_values

count_values("label", q) is aggregate function, which counts the number of points with the same value and stores the counts in a time series with an additional label, which contains each initial value. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

distinct

distinct(q) is aggregate function, which calculates the number of unique values per each group of points with the same timestamp.

geomean

geomean(q) is aggregate function, which calculates geometric mean per each group of points with the same timestamp.

group

group(q) by (group_labels) is aggregate function, which returns 1 per each group_labels for time series returned by q.

This function is supported by PromQL. See also any.

histogram

histogram(q) is aggregate function, which calculates VictoriaMetrics histogram per each group of points with the same timestamp. Useful for visualizing big number of time series via a heatmap. See this article for more details.

limitk

limitk(k, q) by (group_labels) is aggregate function, which returns up to k time series per each group_labels out of time series returned by q. The returned set of time series remain the same across calls.

mad

mad(q) by (group_labels) is aggregate function, which returns the Median absolute deviation per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

See also range_mad, mad_over_time, outliers_mad and stddev.

max

max(q) by (group_labels) is aggregate function, which returns the maximum value per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

median

median(q) by (group_labels) is aggregate function, which returns the median value per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

min

min(q) by (group_labels) is aggregate function, which returns the minimum value per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

mode

mode(q) by (group_labels) is aggregate function, which returns mode per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

outliers_iqr

outliers_iqr(q) is aggregate function, which returns time series from q with at least a single point outside e.g. Interquartile range outlier bounds [q25-1.5*iqr .. q75+1.5*iqr] comparing to other time series at the given point, where:

iqr is an Interquartile range calculated independently per each point on the graph across q series.
q25 and q75 are 25th and 75th percentiles calculated independently per each point on the graph across q series.

The outliers_iqr() is useful for detecting anomalous series in the group of series. For example, outliers_iqr(temperature) by (country) returns per-country series with anomalous outlier values comparing to the rest of per-country series.

outliers_mad

outliers_mad(tolerance, q) is aggregate function, which returns time series from q with at least a single point outside Median absolute deviation (aka MAD) multiplied by tolerance. E.g. it returns time series with at least a single point below median(q) - mad(q) or a single point above median(q) + mad(q).

See also outliers_iqr, outliersk and mad.

outliersk

outliersk(k, q) is aggregate function, which returns up to k time series with the biggest standard deviation (aka outliers) out of time series returned by q.

quantile

quantile(phi, q) by (group_labels) is aggregate function, which calculates phi-quantile per each group_labels for all the time series returned by q. phi must be in the range [0...1]. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL. See also quantiles and histogram_quantile.

quantiles

quantiles("phiLabel", phi1, ..., phiN, q) is aggregate function, which calculates phi*-quantiles for all the time series returned by q and return them in time series with {phiLabel="phi*"} label. phi* must be in the range [0...1]. The aggregate is calculated individually per each group of points with the same timestamp.

stddev

stddev(q) by (group_labels) is aggregate function, which calculates standard deviation per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

stdvar

stdvar(q) by (group_labels) is aggregate function, which calculates standard variance per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

sum

sum(q) by (group_labels) is aggregate function, which returns the sum per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL.

sum2

sum2(q) by (group_labels) is aggregate function, which calculates the sum of squares per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

topk

topk(k, q) is aggregate function, which returns up to k points with the biggest values across all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp.

This function is supported by PromQL. See also bottomk.

topk_avg

topk_avg(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the biggest averages. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, topk_avg(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the biggest averages plus a time series with {job="other"} label with the sum of the remaining series if any.

topk_last

topk_last(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the biggest last values. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, topk_max(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the biggest maximums plus a time series with {job="other"} label with the sum of the remaining series if any.

topk_max

topk_max(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the biggest maximums. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, topk_max(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the biggest maximums plus a time series with {job="other"} label with the sum of the remaining series if any.

topk_median

topk_median(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the biggest medians. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, topk_median(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the biggest medians plus a time series with {job="other"} label with the sum of the remaining series if any.

topk_min

topk_min(k, q, "other_label=other_value") is aggregate function, which returns up to k time series from q with the biggest minimums. If an optional other_label=other_value arg is set, then the sum of the remaining time series is returned with the given label. For example, topk_min(3, sum(process_resident_memory_bytes) by (job), "job=other") would return up to 3 time series with the biggest minimums plus a time series with {job="other"} label with the sum of the remaining series if any.

zscore

zscore(q) by (group_labels) is aggregate function, which returns z-score values per each group_labels for all the time series returned by q. The aggregate is calculated individually per each group of points with the same timestamp. This function is useful for detecting anomalies in the group of related time series.

Subqueries

MetricsQL supports and extends PromQL subqueries. See this article for details. Any rollup function for something other than series selector form a subquery. Nested rollup functions can be implicit thanks to the implicit query conversions. For example, delta(sum(m)) is implicitly converted to delta(sum(default_rollup(m))[1i:1i]), so it becomes a subquery, since it contains default_rollup nested into delta.

VictoriaMetrics performs subqueries in the following way:

It calculates the inner rollup function using the step value from the outer rollup function. For example, for expression max_over_time(rate(http_requests_total[5m])[1h:30s]) the inner function rate(http_requests_total[5m]) is calculated with step=30s. The resulting data points are aligned by the step.
It calculates the outer rollup function over the results of the inner rollup function using the step value passed by Grafana to /api/v1/query_range.

Implicit query conversions

VictoriaMetrics performs the following implicit conversions for incoming queries before starting the calculations:

If lookbehind window in square brackets is missing inside rollup function, then it is automatically set to the following value:
- To step value passed to /api/v1/query_range or /api/v1/query for all the rollup functions except of default_rollup and rate. This value is known as $__interval in Grafana or 1i in MetricsQL. For example, avg_over_time(temperature) is automatically transformed to avg_over_time(temperature[1i]).
- To the max(step, scrape_interval), where scrape_interval is the interval between raw samples for default_rollup and rate functions. This allows avoiding unexpected gaps on the graph when step is smaller than scrape_interval.
All the series selectors, which aren't wrapped into rollup functions, are automatically wrapped into default_rollup function. Examples:
- foo is transformed to default_rollup(foo)
- foo + bar is transformed to default_rollup(foo) + default_rollup(bar)
- count(up) is transformed to count(default_rollup(up)), because count isn't a rollup function - it is aggregate function
- abs(temperature) is transformed to abs(default_rollup(temperature)), because abs isn't a rollup function - it is transform function
If step in square brackets is missing inside subquery, then 1i step is automatically added there. For example, avg_over_time(rate(http_requests_total[5m])[1h]) is automatically converted to avg_over_time(rate(http_requests_total[5m])[1h:1i]).
If something other than series selector is passed to rollup function, then a subquery with 1i lookbehind window and 1i step is automatically formed. For example, rate(sum(up)) is automatically converted to rate((sum(default_rollup(up)))[1i:1i]).

122 KiB Raw Blame History

MetricsQL

MetricsQL features

keep_metric_names

MetricsQL functions

Rollup functions

absent_over_time

aggr_over_time

ascent_over_time

avg_over_time

changes

changes_prometheus

count_eq_over_time

count_gt_over_time

count_le_over_time

count_ne_over_time

count_over_time

decreases_over_time

default_rollup

delta

delta_prometheus

deriv

deriv_fast

descent_over_time

distinct_over_time

duration_over_time

first_over_time

geomean_over_time

histogram_over_time

hoeffding_bound_lower

hoeffding_bound_upper

holt_winters

idelta

ideriv

increase

increase_prometheus

increase_pure

increases_over_time

integrate

irate

lag

last_over_time

lifetime

mad_over_time

max_over_time

median_over_time

min_over_time

mode_over_time

outlier_iqr_over_time

predict_linear

present_over_time

quantile_over_time

quantiles_over_time

range_over_time

rate

rate_over_sum

resets

rollup

rollup_candlestick

rollup_delta

rollup_deriv

rollup_increase

rollup_rate

rollup_scrape_interval

scrape_interval

share_gt_over_time

share_le_over_time

share_eq_over_time

stale_samples_over_time

stddev_over_time

stdvar_over_time

sum_eq_over_time

sum_gt_over_time

sum_le_over_time

sum_over_time

sum2_over_time

timestamp

timestamp_with_name

tfirst_over_time

tlast_change_over_time

122 KiB

Raw Blame History