Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-26 04:10:08 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6,008 Commits 113 Branches 748 Tags 707 MiB
72a0b49330
Commit Graph

480 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
6e364e19ef
app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs 2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin
0d47c23a03
app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K 
There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues
 2022-02-24 04:05:18 +02:00
Aliaksandr Valialkin
62b46007c5
lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes 
This should reduce memory usage under high time series churn rate
 2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin
b1f94f7f0e
app/vmselect/promql: return at most one time series from absent_over_time() in the same way as Prometheus does 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130
 2022-02-12 15:45:09 +02:00
Aliaksandr Valialkin
96b7de6736
app/vmselect/promql: clarify comments on why the right side of if and and operators are executed at first 2022-02-03 00:26:14 +02:00
Aliaksandr Valialkin
4b850c2a59
app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values 
Such filters may slow down time series search, so just skip them.

This is a follow-up for e7f1ceeb84

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827
 2022-02-02 23:40:02 +02:00
Aliaksandr Valialkin
2016a2c899
app/vmselect/promql: properly handle foo or bar queries 
Such queries may miss `bar` results after the commit e7f1ceeb84
because common label filters from `foo` could be mistakenly applied to `bar`.
 2022-02-01 17:40:51 +02:00
Aliaksandr Valialkin
4bdd10ab90
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 
Follow-up for f4989edd96

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin
a8d22e1223
app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first 
PromQL and MetricsQL operators are case-insensitive
 2022-01-31 22:06:15 +02:00
Aliaksandr Valialkin
e7f1ceeb84
app/vmselect/promql: optimize queries, which join on _info metrics. 
Automatically add common filters from one side of binary operation
to the other side before sending the query to storage subsystem.

See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/
and https://www.robustperception.io/exposing-the-software-version-to-prometheus
 2022-01-31 19:32:36 +02:00
Aliaksandr Valialkin
f4989edd96
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions 
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().

Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.

Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-25 15:24:44 +02:00
Aliaksandr Valialkin
dc7b63a793
app/vmselect/promql: properly keep metric names when optimized path is used for aggregate function calculations 
For example, `sum(rate(...) keep_metric_names) by (__name__)` didn't leave the original metric name because of this issue.

This is a follup-up for 1bdc71d917

Udates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949
 2022-01-17 15:30:30 +02:00
Aliaksandr Valialkin
1bdc71d917
app/vmselect/promql: implement keep_metric_names modifier for transform and rollup functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949
 2022-01-14 04:14:59 +02:00
Aliaksandr Valialkin
f41846d002
app/vmselect/promql: add stale_samples_over_time() function 2022-01-14 01:48:04 +02:00
Aliaksandr Valialkin
c883c15878
app/vmselect/promql: add support for @ modifier 
Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier

Extend the support with the following features:
* Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()`
* Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)`
  returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348
 2022-01-13 22:12:06 +02:00
Aliaksandr Valialkin
9469696e46
app/vmselect/promql: fix limit_offset() test 
The test has been broken in addae7fc6a
 2022-01-13 16:59:28 +02:00
Aliaksandr Valialkin
974d9c0eee
app/vmselect/promql: follow-up after 177e345d8a 
* Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions.
* Simplify their implementation
* Mention these functions in docs/CHANGELOG.md
 2021-12-20 13:19:44 +02:00
匠心零度
177e345d8a
add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972) 
Co-authored-by: lirenzuo <lirenzuo@shein.com>
 2021-12-20 12:32:43 +02:00
Aliaksandr Valialkin
5efe377a26
app/vmselect/promql: add timestamp_with_name(m[d]) function 
This function works the same as `timestamp()`, but doesn't remove source time series names.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388
 2021-12-15 23:37:07 +02:00
Aliaksandr Valialkin
d1f8915ed1
app/vmselect/promql: preserve the order of time series passed to limit_offset() function 
Previously time series passed to `limit_offset()` were shuffled according to hash for their labels.
This was unexpected behaviour for most users.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951
 2021-12-12 18:04:58 +02:00
Aliaksandr Valialkin
ff15a752c1
app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863
 2021-12-06 17:07:09 +02:00
Michael Fuller
cf8c171f85
vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838) 
Co-authored-by: Michael Fuller <mfuller@digitalocean.com>
 2021-11-22 13:20:42 +03:00
Aliaksandr Valialkin
9bee043ff2
app/vmselect/promql: consistently return zero from deriv(const) 2021-11-17 18:02:05 +02:00
Aliaksandr Valialkin
b900560b83
app/vmselect/promql: add now() function, which returns the current timestamp as a floating-point value in seconds 2021-11-17 16:35:30 +02:00
Aliaksandr Valialkin
f43586c63c
app/vmselect/promql: arrange function names in the code in alphabetical order 
This should simplify code maintenance in the future
 2021-11-14 13:55:06 +02:00
Aliaksandr Valialkin
9fa098d8e3
app/vmselect/promql: prevent from incorrect calculations for deriv() over multiple samples with identical timestamps 2021-11-12 13:50:43 +02:00
Aliaksandr Valialkin
840ac283ef
app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
 2021-11-09 11:41:56 +02:00
Aliaksandr Valialkin
3419ac1d36
app/vmselect/promql: add duration_over_time(m[d], max_interval) function 
This function calculates the actual lifetime of the time series on the given lookbehdind window `d`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
 2021-11-08 13:14:09 +02:00
Aliaksandr Valialkin
27044b84d2
app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778
 2021-11-03 16:02:27 +02:00
Aliaksandr Valialkin
43a58bd618
app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names 2021-11-03 13:19:08 +02:00
Aliaksandr Valialkin
1952ab99aa
all: fix build issues and tests for Apple M1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653
 2021-10-27 15:06:34 +03:00
Aliaksandr Valialkin
df8f967040
app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738
 2021-10-24 13:31:14 +03:00
Aliaksandr Valialkin
da97e58979
app/vmselect/promql: randomize the static selection of time series returned from limitk() 
Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series.
Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.
 2021-10-16 21:16:49 +03:00
Aliaksandr Valialkin
cae174b11c
app/vmselect/promql: typo fix in comment: didsn't -> didn't 2021-10-16 13:00:34 +03:00
Aliaksandr Valialkin
a5001b9c20
app/vmselect/promql: add atan2 binary operator, which is going to be added in Prometheus 2.31 
See https://github.com/prometheus/prometheus/pull/9248
 2021-10-11 21:15:53 +03:00
Aliaksandr Valialkin
81c6720392
app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31 
See https://github.com/prometheus/prometheus/issues/9233
 2021-10-11 21:01:33 +03:00
Aliaksandr Valialkin
92b92d4d2c
app/vmselect/promql: consistently return the same set of time series from limitk() function 
This is the expected behaviour by most users.
 2021-10-08 19:53:52 +03:00
Aliaksandr Valialkin
2748255c8b
app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow 
The list of functions, which can adjust lookbehind window is more limited than the rest of functions,
so it is better from maintainability and readability PoV using the allowlist instead of blocklist.
 2021-10-07 13:18:42 +03:00
Aliaksandr Valialkin
c45210a6f9
app/vmselect/promql: return back the behaviour for deriv() function when the lookbehind window doesnt contain enough points 
It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d`
contains less than 2 samples in the same way as `rate()` does.

This is a follow-up after 3e084be06b .
 2021-10-07 12:52:27 +03:00
Roman Khavronenko
3e084be06b
app/vmselect: make predict_linear and deriv compatible with Prometheus (#1681) 
Previously, `predict_linear` returned slightly different results comparing
to Prometheus. The change makes linear regression algorithm compatible
with Prometheus.

`deriv` was excluded from the list of functions which can adjust the time
window for the same reasons.
 2021-10-07 12:50:49 +03:00
Aliaksandr Valialkin
0e3de5a0cc
app/vmselect/promql: add topk_last and bottomk_last functions 2021-09-30 13:22:52 +03:00
Roman Khavronenko
a31407006c
app/vmselect: fix binary comparison func (#1667) 
The fix makes the binary comparison func to check for NaNs
before executing the actual comparison. This prevents VM
to return values for non-existing samples for expressions
which contain bool comparisons. Please see added test
for example.
 2021-09-30 12:24:17 +03:00
Roman Khavronenko
344490d89b
app/vmselect: fix testRowsEqual func NaN checks (#1666) 
It appeared, that `testRowsEqual` NaN comparison was incorrect.
The fix caused some tests to fail. Please see the change and
tests updated.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-30 12:08:47 +03:00
Roman Khavronenko
5dc84bf210
app/vmselect: disable time-window adjustment for min/max_over_time funcs (#1658) 
Adjustment results into discrepancy between Prometheus and VM on time windows
smaller than scrape interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:43:21 +03:00
Roman Khavronenko
de810031bf
app/vmselect: always return zero for stddev func if there is only one value (#1659) 
The fix will always return zero if received set of items consists of one
element only, which also means no deviation.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:38:55 +03:00
Roman Khavronenko
dd536b475c
app/vmselect: return NaN instead of 0 for empty value sets (#1660) 
The change affects `count/stddev/stdvar_over_time` funcs and makes
them to return NaN instead of zero when there is no datapoints
in a time window.
This is needed for improving compatibility with Prometheus.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:37:04 +03:00
Roman Khavronenko
03cd93bf1a
app/vmselect: rm quantile_over_time fast-path optimisations (#1662) 
The removed fast path optimisations weren't consistent with
`quantile` function behavior and results into discrepancy.
Specifically, results didn't match in cases when:
* 0 < phi > 1;
* values contain only one element.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:35:14 +03:00
Aliaksandr Valialkin
2efe0acfc9
app/vmselect/promql: add rollup_scrape_interval(m[d]) function 
It calculates the min, max and avg scrape intervals for m over the given lookbehind window d
 2021-09-27 19:21:24 +03:00
Aliaksandr Valialkin
c4c77aa2dd
app/vmselect/promql: follow-up after 526dd93b32 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
 2021-09-27 18:55:39 +03:00
Roman Khavronenko
526dd93b32
app/vmselect: quantile func compatiblity with Prometheus (#1646) 
* app/vmselect: `quantile` func compatiblity with Prometheus

The `quantile` func was previously calculated by https://github.com/valyala/histogram
package. The result of such calculation was always the closest real value to
requested quantile. While in Prometheus implementation interpolation is used.
Such difference may result into discrepancy in output between Prometheus and
VictoriaMetrics.

This commit adds a Prometheus-like `quantile` function. It also used by other
functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: `quantile` review fixes

* quantile functions were split into multiple to provide
different API for already sorted data;
* float64sPool is used for reducing allocations. Items in pool may have
different sizes, but defining a new pool was complicates due to name collisions;

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-27 18:02:41 +03:00
Aliaksandr Valialkin
8ed95e82c6
app/vmselect/promql: follow-up after 57b3320478 2021-09-24 01:24:18 +03:00
Roman Khavronenko
57b3320478
app/vmselect: make sorting for query result similar to Prometheus (#1647) 
* app/vmselect: make sorting for query result similar to Prometheus

Updated sorting allows to get the order of series in result similar or equal
to what Prometheus returns.
The change is needed for compatibility reasons.

* Update app/vmselect/promql/exec_test.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-24 01:03:12 +03:00
Aliaksandr Valialkin
e564411a62
app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus 
Previously VictoriaMetrics could return unexpected result of the right-hand side operand
had multiple time series with the given set of labels mentioned in `on(labels)`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643
 2021-09-24 00:46:25 +03:00
Aliaksandr Valialkin
9a3d0c43b5 app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once 2021-09-17 23:35:10 +03:00
Aliaksandr Valialkin
2951dd0a57 app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function 
This function calculates multiple quantiles over the given buckets at once

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-09-17 13:32:39 +03:00
Aliaksandr Valialkin
5a44be0e52 app/vmselect/promql: optimize quantiles() calculation 
Calculate quantiles in one go instead of calculating each quantile individually
 2021-09-17 12:33:42 +03:00
Aliaksandr Valialkin
e60dfc96ff app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL 2021-09-16 13:33:53 +03:00
Aliaksandr Valialkin
d1a16e0891 app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function 2021-09-13 13:48:06 +03:00
Aliaksandr Valialkin
ea943911bc app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600
 2021-09-09 19:21:18 +03:00
Aliaksandr Valialkin
5ea689d61b app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-08-27 18:37:20 +03:00
Aliaksandr Valialkin
bec18e4fe9 app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570
 2021-08-27 17:15:31 +03:00
Aliaksandr Valialkin
6c5760db9c app/vmselect/promql: make fmt after 0078486ea7 2021-08-23 23:06:00 +03:00
Aliaksandr Valialkin
0078486ea7 app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus 
See https://github.com/prometheus/prometheus/pull/8457 for details.
 2021-08-23 11:45:51 +03:00
Aliaksandr Valialkin
4f3a5742eb app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d]) 
This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d
 2021-08-21 19:56:14 +03:00
Aliaksandr Valialkin
a724229b5d app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval 
Since most users do not expect the overriding of explicitly set `[d]`.
 2021-08-19 10:31:48 +03:00
Aliaksandr Valialkin
cdc372bb98 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries 
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
 2021-08-18 13:59:02 +03:00
Aliaksandr Valialkin
db1e62495b app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541
 2021-08-17 13:21:21 +03:00
Aliaksandr Valialkin
bd14b0887e app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability 2021-08-17 11:01:16 +03:00
Aliaksandr Valialkin
113f0a8a07 app/vmselect/promql: drop staleness marks before calling rollupConfig.Do 
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-15 13:21:10 +03:00
Aliaksandr Valialkin
25997a70f1 Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions" 
This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346.

It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc
 2021-08-15 13:19:16 +03:00
Aliaksandr Valialkin
73d7b568da app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions 
Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-14 12:45:57 +03:00
Aliaksandr Valialkin
4401464c22 all: add support for Prometheus staleness markers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin
e92fde7945 app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
 2021-08-03 16:11:49 +03:00
Aliaksandr Valialkin
8b7917cd81 all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin
05672ffc32 app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series 
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
 2021-07-16 01:44:58 +03:00
Aliaksandr Valialkin
a925d5a3e1 app/vmselect/promql: duration handling improvements in MetricsQL queries 
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
 2021-07-12 17:16:41 +03:00
Aliaksandr Valialkin
cdfae0117a app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect 2021-06-23 14:19:00 +03:00
Aliaksandr Valialkin
83a4db813e app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration 2021-06-18 19:04:42 +03:00
Roman Khavronenko
fb4f758715
promql: fix increase_pure calculation for cases with stale series (#1381) 
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
 2021-06-15 17:37:19 +03:00
John Belmonte
67b17cdd68
spelling fix: synonym (#1363) 2021-06-10 08:32:52 +03:00
Aliaksandr Valialkin
d0dca62026 app/vmselect/promql: typo fix in the comment 2021-06-09 18:34:31 +03:00
Aliaksandr Valialkin
0842bb9294 app/vmselect/promql: add timezone_offset(tz) function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
 2021-05-20 11:53:09 +03:00
Aliaksandr Valialkin
544821b719 app/vmselect/promql: fix tests after d3fa0ccabd 2021-04-08 00:18:01 +03:00
Aliaksandr Valialkin
d3fa0ccabd app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-08 00:09:40 +03:00
Aliaksandr Valialkin
1177dca3da app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-07 14:16:08 +03:00
Aliaksandr Valialkin
7a0b964e8d app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label) 2021-04-03 22:05:06 +03:00
Aliaksandr Valialkin
3055ab0115 app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value") 2021-04-02 23:55:54 +03:00
Aliaksandr Valialkin
c79e4a2f90 app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users 
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
 2021-04-02 15:02:08 +03:00
Aliaksandr Valialkin
aa81039b42 app/vmselect: log the metric which trigger rollup result cache reset 
This should help finding the source of stale metrics
 2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
0b2a66db30 app/vmselect/promql: do not merge time series during requests to /api/v1/query 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141
 2021-03-25 13:56:07 +02:00
Aliaksandr Valialkin
71b72304ae app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag 2021-03-22 16:45:34 +02:00
Aliaksandr Valialkin
cef010d5f7 app/vmselect/promql: increment key prefix for faster reset for rollup result cache 2021-03-22 11:59:07 +02:00
Aliaksandr Valialkin
b1713e3fcd app/vmselect/promql: typo fix after 9666834045 2021-03-17 15:12:11 +02:00
Aliaksandr Valialkin
9666834045 app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function 
This should improve accuracy for the returned buckets
 2021-03-17 14:31:41 +02:00
Aliaksandr Valialkin
f0a4157f89 app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time() 2021-03-16 12:24:21 +02:00
Aliaksandr Valialkin
923cdb0552 app/vmselect/promql: reduce overhead on scrape interval estimation 
It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.
 2021-03-15 20:31:33 +02:00
Aliaksandr Valialkin
82aab87446 app/vmselect/promql: fix tests after 2dae0a2c47 2021-03-15 20:18:59 +02:00
Aliaksandr Valialkin
2dae0a2c47 app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point 2021-03-15 12:36:33 +02:00
Aliaksandr Valialkin
fe8b12fbad app/vmselect/promql: follow up for 433fff0006 2021-03-09 12:48:44 +02:00
Nikolay
433fff0006
duplicate timeseries fix for prometheus_buckets function (#1119) 
* try fix for prometheus_buckets

* merge possible end of the bucket collision
 2021-03-09 12:26:23 +02:00
Aliaksandr Valialkin
a14053ffa0 app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL 2021-03-04 14:12:07 +02:00
Nikolay
186c078fac
adds enforced tag filters into cache key (#1095) 2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin
d86e9b49c4 app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge 
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
 2021-02-26 22:56:36 +02:00
Aliaksandr Valialkin
98854e5f2b app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release 
See https://twitter.com/roidelapluie/status/1363428376162295811

The `last_over_time(m[d])` function already exists in MetricsQL.
 2021-02-24 17:24:56 +02:00
Aliaksandr Valialkin
f4135b0d14 app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does 2021-02-24 00:42:22 +02:00
Aliaksandr Valialkin
fa03e0d210 app/vmselect/promql: add increase_pure() function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
 2021-02-22 19:14:15 +02:00
Aliaksandr Valialkin
9c70c1f21f app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics 2021-02-18 22:07:29 +02:00
Aliaksandr Valialkin
4e39bf148c vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
2d33230793 app/vmselect/promql: properly make copies of EvalConfig 2021-02-11 12:41:15 +02:00
Aliaksandr Valialkin
7a3a9421f3 app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2 
This should prevent from data races if the underlying functions modify EvalConfig contents.
 2021-02-10 23:05:55 +02:00
Aliaksandr Valialkin
04faea8b45 app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel 
This should reduce query execution times.
 2021-02-10 22:59:39 +02:00
Aliaksandr Valialkin
8629fd8a72 app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax 2021-02-03 20:36:01 +02:00
Aliaksandr Valialkin
b43b498fd8 app/vmselect: add ability to pass extra_label=<label>=<value> query arg to Prometheus Querying API 
This enforced `{label="value"}` label filter to the query.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021
 2021-02-01 18:04:17 +02:00
Aliaksandr Valialkin
c164a8d231 app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag 
This should reduce incorrect usage and assumptions for this flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020
 2021-01-22 13:00:10 +02:00
Aliaksandr Valialkin
8749c2dd92 app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time 2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin
1051d8aa2d app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
 2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin
acf1a2c72b app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
 2021-01-13 10:59:42 +02:00
Aliaksandr Valialkin
1837f2f7d3 app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d 2021-01-12 16:12:12 +02:00
Aliaksandr Valialkin
c86286ec1d app/vmselect/promql: do not ajdust offset value provided in the query 
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
 2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin
df0309eae0 app/vmselect/promql: simplify defer call for querystats.RegisterQuery 2020-12-27 12:06:04 +02:00
Aliaksandr Valialkin
59183f66d0 app/vmselect: refactor /api/v1/stats/top_queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
 2020-12-25 16:44:29 +02:00
Nikolay
86630350bf
Adds query stats handler (#945) 
* Adds query stat handler,
for query and query_range api, victoriametrics tracks query execution time,
stats are expored at /api/v1/status/queries endpoint with topN param
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907

* fixed query stats bugs

* improves queryStats tracker

* improves query stat

* small fix

* fix tests

* added more tests

* fixes 386 tests

* naming fixes

* adds drop for outdated records
 2020-12-25 16:42:05 +02:00
Aliaksandr Valialkin
88ac4dfc07 app/vmselect: properly parse negative combined offsets such as -1h2m3s 
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
 2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin
11fa458e39 app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100 2020-12-15 13:40:46 +02:00
Aliaksandr Valialkin
5ebfc275e6 app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested 2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
1906f841c9 app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect 2020-12-08 15:16:17 +02:00
Aliaksandr Valialkin
e2e8ef86d9 app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N 2020-12-05 12:30:46 +02:00
Aliaksandr Valialkin
66379cc69f app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936
 2020-12-03 21:47:36 +02:00
Aliaksandr Valialkin
44a34a0f5f app/vmselect/promql: make fmt 2020-12-02 21:33:35 +02:00
Aliaksandr Valialkin
1982505c2b app/vmselect/promql: return nan from minute(m) when m equals to nan 
This aligns VictoriaMetrics behaviour with Prometheus behaviour.

The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 20:16:58 +02:00
Aliaksandr Valialkin
9d87496b50 app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. 
This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.
 2020-12-02 13:12:50 +02:00
Aliaksandr Valialkin
91a4c279cc app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true 
This aligns MetricsQL behavior to Prometheus' one.

The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 12:09:34 +02:00
Aliaksandr Valialkin
700bda8e2e app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does 2020-12-02 00:28:26 +02:00
Aliaksandr Valialkin
2859a452d4 app/vmselect: add remoteAddr to slow query log in order to improve debuggability 
This will simplify identifying the client that sends slow queries to VictoriaMetrics.
 2020-11-18 20:38:32 +02:00
Aliaksandr Valialkin
266788be14 app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place 
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
 2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
d9d01f976b app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points 
Do not take into account spare data point value if the next point will is located too far from the current point.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 15:23:44 +02:00
Aliaksandr Valialkin
1f19c167a4 app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() 
This should remove unexpected spikes at the end of gaps.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 14:59:24 +02:00
Aliaksandr Valialkin
348edd92fe app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters 2020-11-11 12:39:07 +02:00
Aliaksandr Valialkin
47390d8947 app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2020-11-10 14:51:44 +02:00
Aliaksandr Valialkin
ba4a2c8bca app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum 2020-11-10 01:18:08 +02:00
Aliaksandr Valialkin
dd6bfa50e9 app/vmselect/promql: code cleanup after 43823addea 2020-11-06 01:30:50 +02:00
n4mine
43823addea
app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883) 2020-11-06 01:29:24 +02:00
Aliaksandr Valialkin
5e75c389e6 app/vmselect/promql: allow dropping trailing sample only for default_rollup function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
 2020-11-02 02:10:59 +02:00
Aliaksandr Valialkin
cb71af216a app/vmselect/promql: go fmt 2020-11-02 00:15:29 +02:00
Aliaksandr Valialkin
daacbc7e34 app/vmselect/promql: do not drop trailing datapoints for instant queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-11-02 00:12:37 +02:00
Aliaksandr Valialkin
c539494b36 app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K 2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
28353e48ca app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series 
Now trailing data points are additionally dropped for time series with a single raw sample

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-17 10:44:34 +03:00
Aliaksandr Valialkin
a2b9476897 app/vmselect/promql: return a single time series at max from absent() function like Prometheus does 2020-10-13 15:56:04 +03:00
Aliaksandr Valialkin
9aa3b65766 app/vmselect/promql: improve time series staleness detection 
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-13 12:19:57 +03:00