Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-23 00:46:23 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6,756 Commits 114 Branches 748 Tags 662 MiB
9169f65521
Commit Graph

499 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
26cf680468
app/vmselect/promql: remove memory allocations from sortMetricTags() 2023-01-09 22:22:15 -08:00
Aliaksandr Valialkin
4f0c11ee93
app/vmselect/promql: intern output series names inside timeseriesToResult() 
This reduces the number of memory allocations for repeated queries,
which return (almost) the same set of time series.
 2023-01-09 22:19:56 -08:00
Aliaksandr Valialkin
562d6bca08
app/vmselect/promql: intern output series names during normal aggregation 2023-01-09 22:15:24 -08:00
Aliaksandr Valialkin
21ee9a1fab
app/vmselect/promql: intern output series names during incremental aggregation 
This should reduce the number of memory allocations for repeated queries
 2023-01-09 22:11:36 -08:00
Aliaksandr Valialkin
04dff34de4
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0 
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
 2023-01-05 01:50:10 -08:00
Aliaksandr Valialkin
f1441a598f
app/vmselect/promql: add tests for d3de110070 2022-12-21 20:25:21 -08:00
Aliaksandr Valialkin
d3de110070
app/vmselect/promql: make sure that label_replace() doesn't create an empty dst_label if the src_label doesn't match regex 2022-12-21 20:20:01 -08:00
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly 
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
 2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
 2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Aliaksandr Valialkin
d9c3a2b605
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range 
This may be useful for analyzing correlation between time series with different value ranges
 2022-11-21 23:25:00 +02:00
Aliaksandr Valialkin
2c9e403d5f
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
 2022-11-21 16:47:37 +02:00
Aliaksandr Valialkin
b796a0dc3f
app/vmselect/promql: optimize e1 op e2 when e1 returns an empty result 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
 2022-11-21 16:09:10 +02:00
Aliaksandr Valialkin
a260e2659e
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:40 +02:00
Aliaksandr Valialkin
c1a3192d8b
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:38:48 +02:00
Aliaksandr Valialkin
c4265322f4
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin
a10647e0bf
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:13:47 +03:00
Aliaksandr Valialkin
e6fd33044f
app/vmselect/promql: follow-up for 930f1ee153 
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
 2022-10-13 12:03:08 +03:00
Siqi Liu
930f1ee153
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225) 
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
 2022-10-13 11:57:16 +03:00
Aliaksandr Valialkin
b8da90b893
app/vmselect/promql: properly handle zero and negative values for -search.maxMemoryPerQuery 
This is a follow-up for 04a05f161c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-12 09:25:17 +03:00
Aliaksandr Valialkin
04a05f161c
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set 
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-10 21:45:13 +03:00
Aliaksandr Valialkin
5138eaeea0
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-08 01:08:05 +03:00
Aliaksandr Valialkin
f452c84579
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:22:24 +03:00
Aliaksandr Valialkin
40e899fd67
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:21 +03:00
Aliaksandr Valialkin
fb1cc3cc94
app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores 
Use sync.Map instead of a global mutex there. This should lift scalability limits
on systems with many CPU cores.
 2022-10-01 20:00:03 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function 
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
 2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178) 
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-09-30 08:20:34 +03:00
Roman Khavronenko
166d444159
vmselect/rollup: rm workaround for slow-changing counters (#3163) 
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-09-26 15:33:25 +03:00
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982) 
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-14 17:41:09 +03:00
Dmytro Kozlov
4415c71a2b
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049) 
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
 2022-09-06 13:25:59 +03:00
Aliaksandr Valialkin
7dc632719d
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
 2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin
4076277cf0
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance 
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
 2022-09-02 19:46:27 +03:00
Aliaksandr Valialkin
ad11b8d83d
app/vmselect/promql: follow-up after 2d71b4859c 
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
 2022-08-27 01:35:49 +03:00
Dmytro Kozlov
2d71b4859c
vmselect/promql: fix panic in histogram_quantiles function (#3029) 
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
 2022-08-27 01:33:56 +03:00
Dmytro Kozlov
463ea6897b
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963) 
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-08-24 15:25:18 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
4ac79d29ad
app/vmselect: follow-up after 63e0f16062 
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
 2022-08-15 13:50:16 +03:00
Roman Khavronenko
63e0f16062
vmselect: introduce UserReadableError type of error (#2894) 
When read query fails, VM returns rich error message with
all the details. While these details might be useful
for debugging specific cases, they're usually too verbose
for users.
Introducing a new error type `UserReadableError` is supposed
to allow to return to user only the most important parts
of the error trace. This supposed to improve error readability
in web interfaces such as VMUI or Grafana.

The full error trace is still logged with the full context
and can be found in vmselect logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-08-15 13:38:47 +03:00
Dmytro Kozlov
a927814e7b
vmselect/promql: add tests for vmrangeBucketsToLE (#2907) 
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-26 20:42:41 +03:00
Aliaksandr Valialkin
c888e6b9be
app/vmselect/promql: reduce the diff for f148cffc8a 
This is a follow-up for c826f06366
 2022-07-26 19:20:48 +03:00
Alan Liang
c826f06366
vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902) 
Co-authored-by: alanwzliang <alanwzliang@tencent.com>
 2022-07-25 10:55:13 +03:00
Roman Khavronenko
9ccf695d57
vmselect: return correct error for second part of expression (#2893) 
Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-07-20 16:44:28 +02:00
Aliaksandr Valialkin
cc7d499bbd
app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied 
This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886
 2022-07-19 14:27:48 +03:00
Aliaksandr Valialkin
b5da47bfaf
app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing 2022-07-18 14:24:54 +03:00
Aliaksandr Valialkin
0792c4ca90
app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results 2022-07-18 14:24:53 +03:00
Aliaksandr Valialkin
29e53b9f55
app/vmselect/promql: consistency update after 93fbd0c54b 2022-07-13 12:33:14 +03:00
Roman Khavronenko
93fbd0c54b
promql: return step as scrapeInterval when it can't be calculated (#2865) 
The change allows to specify default value for `getScrapeInterval`
function when actual interval can't be calculated.

Before the change, function were returning `maxSilenceInterval` (5m)
in such cases, which may be not correct for instant queries processing.
The specific scenario where using `maxSilenceInterval` caused issues
is the following:
1. Series becomes stale;
2. Client (in this case vmalert) continues to request series every 15s;
3. Database returns empty results as expected;
4. But at some specific moment of time database returns datapoints from `now()-5m`,
because lookback window was extended to `maxSilenceInterval`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-07-13 12:27:38 +03:00
Aliaksandr Valialkin
c2197ad139
app/vmselect/promql: validate function name before evaluating its arguments 
This avoids unneeded evaluation of args for unknown functions
 2022-07-12 19:48:26 +03:00
Roman Khavronenko
e1a41cfab5
metricsql: properly evaluate timezone_offset over time interval (#2842) 
* metricsql: properly evaluate `timezone_offset` over time interval

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-08 14:03:56 +03:00
Aliaksandr Valialkin
ae80cf76e0
app/vmselect: make fmt after f3ece83e67 2022-07-05 14:35:24 +03:00
Aliaksandr Valialkin
f3ece83e67
app/vmselect/promql: properly calculate histogram_quantile over unexpected le buckets 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819
 2022-07-05 13:19:24 +03:00
Aliaksandr Valialkin
84e373e5c7
app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions 2022-06-30 22:39:38 +03:00
Aliaksandr Valialkin
3e2dd85f7d
all: readability improvements for query traces 
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
 2022-06-30 18:20:33 +03:00
Aliaksandr Valialkin
a14188dd8e
app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload 
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792
 2022-06-28 20:18:13 +03:00
Aliaksandr Valialkin
a43f2d0bc5
app/vmselect/promql: show the number of scanned samples in the query trace 2022-06-28 19:26:17 +03:00
Aliaksandr Valialkin
e578549b8a
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day 2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin
a963b2a0aa
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds 2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
e6ed92529b
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This package already has the same name, so there is no need in explicit name
 2022-06-21 20:23:32 +03:00
Aliaksandr Valialkin
12ac255dae
lib/querytracer: make it easier to use by passing trace context message to New and NewChild 
The context message can be extended by calling Donef.
If there is no need to extend the message, then just call Done.
 2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin
41958ed5dd
all: add initial support for query tracing 
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
 2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin
a7f18f8cb2
app/vmselect/promql: do not return values from label_value() if the original time series has no values at the selected timestamps 2022-05-09 17:57:39 +03:00
Aliaksandr Valialkin
cce1b6d7f9
app/vmselect/promql: add tlast_change_over_time(m[d]) function, which returns the timestamp for the last change of m on the given lookbehind window d 2022-04-27 10:59:03 +03:00
Aliaksandr Valialkin
25fe83577d
app/vmselect/promql: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries 
Previously `vector` time series could be unexpectedly returned from such queries
 2022-04-21 15:34:36 +03:00
Aliaksandr Valialkin
d1a9fac894
app/vmselect/promql: fix comparison to nan 
The comparison to nan has been broken in d335cc886c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
 2022-04-21 14:55:37 +03:00
Aliaksandr Valialkin
de892239a9
app/vmselect/promql: add drop_common_labels() function 2022-04-21 14:20:20 +03:00
Aliaksandr Valialkin
7a44ba1234
app/vmselect/promql: fix q default b where b may have empty time series 2022-04-21 11:42:42 +03:00
Aliaksandr Valialkin
d335cc886c
app/vmselect/promql: fix duplicate time series error on joins against time series filtered by values 
This should prevent from `duplicate time series` errors when executing the following query:

kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1)

where `kube_pod_status_phase{phase=~"Pending|Running"}==1` filters out diplicate time series
 2022-04-20 22:18:44 +03:00
Aliaksandr Valialkin
ed97908ca9
app/vmselect/promql: rename removeNaNs() to more clear removeEmptySeries() 2022-04-20 19:53:46 +03:00
Aliaksandr Valialkin
61c7f6beae
app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2406
 2022-04-11 12:34:43 +03:00
Aliaksandr Valialkin
6e364e19ef
app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs 2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin
0d47c23a03
app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K 
There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues
 2022-02-24 04:05:18 +02:00
Aliaksandr Valialkin
62b46007c5
lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes 
This should reduce memory usage under high time series churn rate
 2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin
b1f94f7f0e
app/vmselect/promql: return at most one time series from absent_over_time() in the same way as Prometheus does 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130
 2022-02-12 15:45:09 +02:00
Aliaksandr Valialkin
96b7de6736
app/vmselect/promql: clarify comments on why the right side of if and and operators are executed at first 2022-02-03 00:26:14 +02:00
Aliaksandr Valialkin
4b850c2a59
app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values 
Such filters may slow down time series search, so just skip them.

This is a follow-up for e7f1ceeb84

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827
 2022-02-02 23:40:02 +02:00
Aliaksandr Valialkin
2016a2c899
app/vmselect/promql: properly handle foo or bar queries 
Such queries may miss `bar` results after the commit e7f1ceeb84
because common label filters from `foo` could be mistakenly applied to `bar`.
 2022-02-01 17:40:51 +02:00
Aliaksandr Valialkin
4bdd10ab90
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 
Follow-up for f4989edd96

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin
a8d22e1223
app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first 
PromQL and MetricsQL operators are case-insensitive
 2022-01-31 22:06:15 +02:00
Aliaksandr Valialkin
e7f1ceeb84
app/vmselect/promql: optimize queries, which join on _info metrics. 
Automatically add common filters from one side of binary operation
to the other side before sending the query to storage subsystem.

See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/
and https://www.robustperception.io/exposing-the-software-version-to-prometheus
 2022-01-31 19:32:36 +02:00
Aliaksandr Valialkin
f4989edd96
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions 
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().

Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.

Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-25 15:24:44 +02:00
Aliaksandr Valialkin
dc7b63a793
app/vmselect/promql: properly keep metric names when optimized path is used for aggregate function calculations 
For example, `sum(rate(...) keep_metric_names) by (__name__)` didn't leave the original metric name because of this issue.

This is a follup-up for 1bdc71d917

Udates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949
 2022-01-17 15:30:30 +02:00
Aliaksandr Valialkin
1bdc71d917
app/vmselect/promql: implement keep_metric_names modifier for transform and rollup functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949
 2022-01-14 04:14:59 +02:00
Aliaksandr Valialkin
f41846d002
app/vmselect/promql: add stale_samples_over_time() function 2022-01-14 01:48:04 +02:00
Aliaksandr Valialkin
c883c15878
app/vmselect/promql: add support for @ modifier 
Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier

Extend the support with the following features:
* Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()`
* Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)`
  returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348
 2022-01-13 22:12:06 +02:00
Aliaksandr Valialkin
9469696e46
app/vmselect/promql: fix limit_offset() test 
The test has been broken in addae7fc6a
 2022-01-13 16:59:28 +02:00
Aliaksandr Valialkin
974d9c0eee
app/vmselect/promql: follow-up after 177e345d8a 
* Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions.
* Simplify their implementation
* Mention these functions in docs/CHANGELOG.md
 2021-12-20 13:19:44 +02:00
匠心零度
177e345d8a
add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972) 
Co-authored-by: lirenzuo <lirenzuo@shein.com>
 2021-12-20 12:32:43 +02:00
Aliaksandr Valialkin
5efe377a26
app/vmselect/promql: add timestamp_with_name(m[d]) function 
This function works the same as `timestamp()`, but doesn't remove source time series names.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388
 2021-12-15 23:37:07 +02:00
Aliaksandr Valialkin
d1f8915ed1
app/vmselect/promql: preserve the order of time series passed to limit_offset() function 
Previously time series passed to `limit_offset()` were shuffled according to hash for their labels.
This was unexpected behaviour for most users.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951
 2021-12-12 18:04:58 +02:00
Aliaksandr Valialkin
ff15a752c1
app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863
 2021-12-06 17:07:09 +02:00
Michael Fuller
cf8c171f85
vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838) 
Co-authored-by: Michael Fuller <mfuller@digitalocean.com>
 2021-11-22 13:20:42 +03:00
Aliaksandr Valialkin
9bee043ff2
app/vmselect/promql: consistently return zero from deriv(const) 2021-11-17 18:02:05 +02:00
Aliaksandr Valialkin
b900560b83
app/vmselect/promql: add now() function, which returns the current timestamp as a floating-point value in seconds 2021-11-17 16:35:30 +02:00
Aliaksandr Valialkin
f43586c63c
app/vmselect/promql: arrange function names in the code in alphabetical order 
This should simplify code maintenance in the future
 2021-11-14 13:55:06 +02:00
Aliaksandr Valialkin
9fa098d8e3
app/vmselect/promql: prevent from incorrect calculations for deriv() over multiple samples with identical timestamps 2021-11-12 13:50:43 +02:00
Aliaksandr Valialkin
840ac283ef
app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
 2021-11-09 11:41:56 +02:00
Aliaksandr Valialkin
3419ac1d36
app/vmselect/promql: add duration_over_time(m[d], max_interval) function 
This function calculates the actual lifetime of the time series on the given lookbehdind window `d`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
 2021-11-08 13:14:09 +02:00
Aliaksandr Valialkin
27044b84d2
app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778
 2021-11-03 16:02:27 +02:00
Aliaksandr Valialkin
43a58bd618
app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names 2021-11-03 13:19:08 +02:00
Aliaksandr Valialkin
1952ab99aa
all: fix build issues and tests for Apple M1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653
 2021-10-27 15:06:34 +03:00