Commit Graph

215 Commits

Author SHA1 Message Date
Hui Wang
dd7dd0b1db
metricsql: fix label_join() when dst_label is equal to one of the `… (#5886)
* metricsql: fix label_join() when `dst_label` is equal to one of the `src_label`

* Update app/vmselect/promql/transform.go

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 16:00:22 +02:00
Aliaksandr Valialkin
35f592a02c
app/vmselect/promql: properly handle args in count_values_over_time() function
Prevsiously they were swapped - the first arg should be the label name and the second arg should be label filters

This is a follow-up for e389b7b959e8144fdff5075bf7a5a39b2b0c6dd3

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847
2024-02-25 01:48:18 +02:00
Aliaksandr Valialkin
f7c3dee1c3
app/vmselect/promql: add count_values_over_time() MetricsQL function
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847
2024-02-22 13:39:29 +02:00
Aliaksandr Valialkin
d6e22f2888
app/vmselect: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions to MetricsQL
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641
2024-02-13 23:40:07 +02:00
Aliaksandr Valialkin
ec0ca8e7eb
app/vmselect/promql: really keep metric names when keep_metric_names modifier is applied to binary operator
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556
2024-01-31 02:32:55 +02:00
Hui Wang
fad212c39c
app/vmselect/promql: properly handle possible negative results caused… (#5608)
* app/vmselect/promql: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase()

* fix test
2024-01-21 02:53:29 +02:00
Aliaksandr Valialkin
388d020b7c
app/vmselect/promql: follow-up for ce4f26db02
- Document the bugfix at docs/CHANGELOG.md
- Filter out NaN values before sorting as suggested at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509#discussion_r1447369218
- Revert unrelated changes in lib/filestream and lib/fs
- Use simpler test at app/vmselect/promql/exec_test.go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506
2024-01-16 02:55:08 +02:00
Aliaksandr Valialkin
190a6565ae
app/vmselect/promql: consistently sort results of a or b query
Previously the order of results returned from `a or b` query could change with each request
because the sorting for such query has been disabled in order to satisfy
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763 .

This commit executes `a or b` query as `sortByMetricName(a) or sortByMetricName(b)`.
This makes the order of returned time series consistent across requests,
while maintaining the requirement from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763 ,
e.g. `b` results are consistently put after `a` results.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5393
2024-01-16 01:30:10 +02:00
luckyxiaoqiang
d7897e0d70
app/vmselect/promql: add day_of_year() function (#5368)
Co-authored-by: dingxiaoqiang <dingxiaoqiang@bytedance.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-11-28 11:54:00 +01:00
Aliaksandr Valialkin
ea81f6fc36
app/vmselect/promql: add outliers_iqr(q) and outlier_iqr_over_time(m[d]) functions
These functions allow detecting anomalies in series and samples using Interquartile range method.
See Outliers section at https://en.wikipedia.org/wiki/Interquartile_range for more details.
2023-10-31 22:10:31 +01:00
Aliaksandr Valialkin
da77f4deeb
app/vmselect/promql: add labels_equal(q, "label1", "label2", ...) function
This function returns q series, which have identical values for the listed labels
"label1", "label2", ...

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148
2023-10-16 21:50:11 +02:00
Aliaksandr Valialkin
bdb743c88d
app/vmselect/promql: add drop_empty_series() function for dropping empty series before performing additional calculations
This can be useful in the following queries:

   drop_empty_series(temperature <= 30) default 40

This query drops temperature series with all the values bigger than 30 on the selected time range,
while replacing gaps in the remaining series with 40.

The query without drop_empty_series:

  (temperature <= 30) default 40

would leave all the temperature series with all the values bigger than 30 on the selected time range,
and replace all their values with 40. This is not what could be epxected in some cases
like here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071
2023-10-16 20:44:56 +02:00
Aliaksandr Valialkin
71668637ce
app/vmselect/promql: follow-up for 896c85a4a4
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that bitmap_*(X, NaN) returns NaN

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5021
2023-10-02 20:08:26 +02:00
Dmytro Kozlov
896c85a4a4
app/vmselect: fix bitmap_*() functions behavior (#5021)
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-09-29 12:03:01 +02:00
Aliaksandr Valialkin
e453069dcd
app/vmselect/promql: run make fmt after 3b9605dba5 2023-09-25 16:16:14 +02:00
Aliaksandr Valialkin
3b9605dba5
app/vmselect/promql: do not sort q1 or q2 results
This makes sure that `q2` series are returned after `q1` series in the same way as Prometheus does

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763
2023-09-25 16:14:16 +02:00
Aliaksandr Valialkin
a740159541
app/vmselect/promql: completely substitute median_over_time() WITH template with regular median_over_time() rollup function
This is a follow-up for 34d7a670d0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034
2023-09-25 15:28:12 +02:00
Dmytro Kozlov
d5f9619984
vmagent: add validation of MetricsQL functions (#4991)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-09-15 13:15:23 +02:00
Aliaksandr Valialkin
1c0e065216
app/vmselect/promql: add support for _ delimiters in numeric values
For example, 1_234_567_890 is equivalent to 1234567890,
while 1.234_567_890 is equivalent to 1.234567890
2023-08-30 14:33:41 +02:00
Damon07
3f6efab6ae
{app/vmselect,docs}: support share_eq_over_time#4441 (#4725)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4441

Co-authored-by: wangm <wangmm@tuya.com>
2023-07-31 15:23:59 +02:00
Aliaksandr Valialkin
bd95341190
app/vmselect/promql: fix tests after 781947a7e2 2023-07-20 21:25:38 -07:00
Aliaksandr Valialkin
8815080030
app/vmselect/promql: add the ability to copy all the labels from one side of group_left()/group_right() operation
This is performed by specifying `*` inside group_left()/group_right().
Also allow specifying prefix for the copied labels via `group_left(...) prefix "..."` and `group_right(...) prefix "..."` syntax.
For example, the following query adds all the namespace-related labels to pod info, and prefixes all the copied label names with "ns_" prefix:

  kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels

This resolves the following StackOverflow questions:

- https://stackoverflow.com/questions/76661818/how-to-add-namespace-labels-to-pod-labels-in-prometheus
- https://stackoverflow.com/questions/76653997/how-can-i-make-a-new-copy-of-kube-namespace-labels-metric-with-a-different-name
2023-07-17 19:07:39 -07:00
Aliaksandr Valialkin
be31bdc88c
app/vmselect/promql: recommend to use (a op b) keep_metric_names instead of a op b keep_metric_names
The `a op b keep_metric_names` is ambigouos to `a op (b keep_metric_names)` when `b` is a transform or rollup function.
For example, `a + rate(b) keep_metric_names`. So it is better to use more clear syntax: `(a op b) keep_metric_names`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3710
2023-07-16 23:46:34 -07:00
Zakhar Bessarab
e2367b6d1c
metricsql: add support of using keep_metric_names for binary operations (#4109)
* metricsql: add support of using keep_metric_names for binary operations

This should help to avoid confusion with queries like one in the issue #3710.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-16 03:00:39 -07:00
Aliaksandr Valialkin
4cb024d8a3
all: add support for or filters in series selectors
This commit adds ability to select series matching distinct filters via a single series selector.
For example, the following selector selects series with either {env="prod",job="a"}
or {env="dev",job="b"} labels:

  {env="prod",job="a" or env="dev",job="b"}

The `or` filter is supported in all the VictoriaMetrics tools now.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997
Uses https://github.com/VictoriaMetrics/metricsql/pull/14
2023-07-16 00:06:33 -07:00
Roman Khavronenko
dac21d874b
metricsql: support optional 2nd argument for rollup functions (#3841)
* metricsql: support optional 2nd argument for rollup functions

Support optional 2nd argument `min`, `max` or `avg` for rollup functions:
 * rollup
 * rollup_delta
 * rollup_deriv
 * rollup_increase
 * rollup_rate
 * rollup_scrape_interval

 If second argument is passed, then rollup function will return only the selected aggregation type.
 This change can be useful for situations where only one type of rollup calculation is needed.
 For example, `rollup_rate(requests_total[5m], "max")`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 13:47:52 -08:00
Alexander Marshalov
aaef0bac00
fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-22 18:38:43 -08:00
Aliaksandr Valialkin
2cea923ff7
app/vmselect/promql: add share(q) aggregate function for normalizing results across multiple time series in [0..1] value range per each timestamp and aggregation group 2023-02-18 22:42:01 -08:00
Aliaksandr Valialkin
c86f1f1d1b
app/vmselect/promql: add range_zscore(q) and range_trim_zscore(z, q) functions
These functions may be useful for dropping outliers at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 22:41:59 -08:00
Aliaksandr Valialkin
406822a16c
app/vmselect/promql: add range_mad(q) and range_trim_outliers(k, q) functions
These functions may help trimming outliers during query time
for the use case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 15:19:10 -08:00
Aliaksandr Valialkin
b11bdc46be
app/vmselect/promql: add mad_over_time(m[d]) function
See https://github.com/prometheus/prometheus/issues/5514
2023-02-11 01:06:20 -08:00
Aliaksandr Valialkin
0c625185cb
app/vmselect/promql: updates tests for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664 2023-01-17 23:25:45 -08:00
Aliaksandr Valialkin
04dff34de4
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-01-05 01:50:10 -08:00
Aliaksandr Valialkin
f1441a598f
app/vmselect/promql: add tests for d3de110070 2022-12-21 20:25:21 -08:00
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Aliaksandr Valialkin
d9c3a2b605
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range
This may be useful for analyzing correlation between time series with different value ranges
2022-11-21 23:25:00 +02:00
Aliaksandr Valialkin
2c9e403d5f
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
2022-11-21 16:47:37 +02:00
Aliaksandr Valialkin
a260e2659e
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:40 +02:00
Aliaksandr Valialkin
c1a3192d8b
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:38:48 +02:00
Aliaksandr Valialkin
e6fd33044f
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:03:08 +03:00
Aliaksandr Valialkin
40e899fd67
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:21 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:20:34 +03:00
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:41:09 +03:00
Aliaksandr Valialkin
ad11b8d83d
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:35:49 +03:00
Dmytro Kozlov
2d71b4859c
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:33:56 +03:00
Dmytro Kozlov
463ea6897b
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:25:18 +03:00
Dmytro Kozlov
a927814e7b
vmselect/promql: add tests for vmrangeBucketsToLE (#2907)
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-26 20:42:41 +03:00