Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-22 16:36:27 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
7,462 Commits 114 Branches 748 Tags 655 MiB
825cfdb5ef
Commit Graph

537 Commits

Author SHA1 Message Date
Roman Khavronenko
f46f847531
app/vmselect: always return zero for stddev func if there is only one value (#1659) 
The fix will always return zero if received set of items consists of one
element only, which also means no deviation.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:42:19 +03:00
Roman Khavronenko
72468c9d75
app/vmselect: return NaN instead of 0 for empty value sets (#1660) 
The change affects `count/stddev/stdvar_over_time` funcs and makes
them to return NaN instead of zero when there is no datapoints
in a time window.
This is needed for improving compatibility with Prometheus.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:38:24 +03:00
Roman Khavronenko
c8ac8f44b2
app/vmselect: rm quantile_over_time fast-path optimisations (#1662) 
The removed fast path optimisations weren't consistent with
`quantile` function behavior and results into discrepancy.
Specifically, results didn't match in cases when:
* 0 < phi > 1;
* values contain only one element.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-29 00:35:59 +03:00
Aliaksandr Valialkin
4b3951fd86
app/vmselect/promql: add rollup_scrape_interval(m[d]) function 
It calculates the min, max and avg scrape intervals for m over the given lookbehind window d
 2021-09-27 19:21:55 +03:00
Aliaksandr Valialkin
eff31c10ec
app/vmselect/promql: follow-up after 526dd93b32 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
 2021-09-27 18:57:03 +03:00
Roman Khavronenko
6061464d80
app/vmselect: quantile func compatiblity with Prometheus (#1646) 
* app/vmselect: `quantile` func compatiblity with Prometheus

The `quantile` func was previously calculated by https://github.com/valyala/histogram
package. The result of such calculation was always the closest real value to
requested quantile. While in Prometheus implementation interpolation is used.
Such difference may result into discrepancy in output between Prometheus and
VictoriaMetrics.

This commit adds a Prometheus-like `quantile` function. It also used by other
functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: `quantile` review fixes

* quantile functions were split into multiple to provide
different API for already sorted data;
* float64sPool is used for reducing allocations. Items in pool may have
different sizes, but defining a new pool was complicates due to name collisions;

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2021-09-27 18:57:02 +03:00
Aliaksandr Valialkin
a746fe8778
app/vmselect/promql: follow-up after 57b3320478 2021-09-24 01:25:04 +03:00
Roman Khavronenko
7ac0559d4f
app/vmselect: make sorting for query result similar to Prometheus (#1647) 
* app/vmselect: make sorting for query result similar to Prometheus

Updated sorting allows to get the order of series in result similar or equal
to what Prometheus returns.
The change is needed for compatibility reasons.

* Update app/vmselect/promql/exec_test.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-24 01:25:03 +03:00
Aliaksandr Valialkin
1115c2f235
app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus 
Previously VictoriaMetrics could return unexpected result of the right-hand side operand
had multiple time series with the given set of labels mentioned in `on(labels)`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643
 2021-09-24 00:56:35 +03:00
Aliaksandr Valialkin
76c650bbda app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once 2021-09-17 23:35:04 +03:00
Aliaksandr Valialkin
81d24045c2 app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function 
This function calculates multiple quantiles over the given buckets at once

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-09-17 13:39:06 +03:00
Aliaksandr Valialkin
4ed7548ced app/vmselect/promql: optimize quantiles() calculation 
Calculate quantiles in one go instead of calculating each quantile individually
 2021-09-17 12:38:21 +03:00
Aliaksandr Valialkin
3f8de4f82f app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL 2021-09-16 13:35:38 +03:00
Aliaksandr Valialkin
19516aedb5 app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function 2021-09-13 13:48:18 +03:00
Aliaksandr Valialkin
9b07cb2988 app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600
 2021-09-09 19:24:10 +03:00
Aliaksandr Valialkin
39bb6bdd79 app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-08-27 18:40:25 +03:00
Aliaksandr Valialkin
335de30083 app/vmselect/promql: make fmt after 0078486ea7 2021-08-23 23:05:34 +03:00
Aliaksandr Valialkin
40b06e84f8 app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus 
See https://github.com/prometheus/prometheus/pull/8457 for details.
 2021-08-23 11:46:29 +03:00
Aliaksandr Valialkin
91534057a3 app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d]) 
This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d
 2021-08-21 19:56:56 +03:00
Aliaksandr Valialkin
ee1f3414d1 app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval 
Since most users do not expect the overriding of explicitly set `[d]`.
 2021-08-19 10:33:10 +03:00
Aliaksandr Valialkin
5d92fafc40 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries 
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
 2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin
49886ecbc8 app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541
 2021-08-17 13:22:15 +03:00
Aliaksandr Valialkin
38065bec7b app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability 2021-08-17 11:00:58 +03:00
Aliaksandr Valialkin
5420c3d967 app/vmselect/promql: drop staleness marks before calling rollupConfig.Do 
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-15 13:22:26 +03:00
Aliaksandr Valialkin
6c4c54eaad Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions" 
This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346.

It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc
 2021-08-15 13:22:24 +03:00
Aliaksandr Valialkin
af4a306d7b app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions 
Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-14 12:45:31 +03:00
Aliaksandr Valialkin
c1f81f08d4 all: add support for Prometheus staleness markers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin
60cfa5f100 app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
 2021-08-03 12:21:53 +03:00
Aliaksandr Valialkin
5d255846ac all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin
b047feeb8b app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series 
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
 2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin
9add9d86a6 app/vmselect/promql: duration handling improvements in MetricsQL queries 
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
 2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin
856aecae05 app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect 2021-06-23 14:18:37 +03:00
Aliaksandr Valialkin
b92d110cad app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration 2021-06-18 19:07:03 +03:00
Roman Khavronenko
a15c947045 promql: fix increase_pure calculation for cases with stale series (#1381) 
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
 2021-06-15 17:37:51 +03:00
John Belmonte
3e79f3994e spelling fix: synonym (#1363) 2021-06-11 10:58:48 +03:00
Aliaksandr Valialkin
e8e7f03394 app/vmselect/promql: typo fix in the comment 2021-06-09 18:34:57 +03:00
Aliaksandr Valialkin
180829b8c2 app/vmselect/promql: add timezone_offset(tz) function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
 2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin
56b08390f6 app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode 
`vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.
 2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin
e48fd5776b app/vmselect/promql: fix tests after d3fa0ccabd 2021-04-08 00:17:40 +03:00
Aliaksandr Valialkin
f90bf265f4 app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-08 00:10:33 +03:00
Aliaksandr Valialkin
3d5f1f779f app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-07 14:16:02 +03:00
Aliaksandr Valialkin
4aa59cae7c app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label) 2021-04-03 22:06:50 +03:00
Aliaksandr Valialkin
b9469de410 app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value") 2021-04-02 23:56:30 +03:00
Aliaksandr Valialkin
31d3d02d25 app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users 
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
 2021-04-02 16:50:02 +03:00
Aliaksandr Valialkin
b473c21915 app/vmselect/promql: do not merge time series during requests to /api/v1/query 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141
 2021-03-25 13:56:23 +02:00
Aliaksandr Valialkin
7f9863254d app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag 2021-03-22 16:44:36 +02:00
Aliaksandr Valialkin
92531a38c4 app/vmselect/promql: increment key prefix for faster reset for rollup result cache 2021-03-22 11:59:39 +02:00
Aliaksandr Valialkin
667d129e1e app/vmselect/promql: typo fix after 9666834045 2021-03-17 15:11:54 +02:00
Aliaksandr Valialkin
69201806f8 app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function 
This should improve accuracy for the returned buckets
 2021-03-17 14:31:29 +02:00
Aliaksandr Valialkin
f54ece438d app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time() 2021-03-16 12:23:55 +02:00
Aliaksandr Valialkin
30ab0eec27 app/vmselect/promql: reduce overhead on scrape interval estimation 
It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.
 2021-03-15 20:32:35 +02:00
Aliaksandr Valialkin
8844b38745 app/vmselect/promql: fix tests after 2dae0a2c47 2021-03-15 20:20:27 +02:00
Aliaksandr Valialkin
6b9bba7448 app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point 2021-03-15 12:38:59 +02:00
Aliaksandr Valialkin
942890b1bb app/vmselect/promql: follow up for 433fff0006 2021-03-09 12:54:23 +02:00
Nikolay
28e450cd7c duplicate timeseries fix for prometheus_buckets function (#1119) 
* try fix for prometheus_buckets

* merge possible end of the bucket collision
 2021-03-09 12:54:21 +02:00
Aliaksandr Valialkin
5277507932 app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL 2021-03-04 14:12:52 +02:00
Nikolay
673b10dd7f adds enforced tag filters into cache key (#1095) 2021-02-27 00:23:38 +02:00
Aliaksandr Valialkin
e38ca28d99 app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge 
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
 2021-02-26 22:56:04 +02:00
Aliaksandr Valialkin
0144b164c7 app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release 
See https://twitter.com/roidelapluie/status/1363428376162295811

The `last_over_time(m[d])` function already exists in MetricsQL.
 2021-02-24 17:26:27 +02:00
Aliaksandr Valialkin
d16effc29e app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does 2021-02-24 00:41:30 +02:00
Aliaksandr Valialkin
5dbe88a1c6 app/vmselect/promql: add increase_pure() function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
 2021-02-22 19:15:31 +02:00
Aliaksandr Valialkin
bd1d906eee app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics 2021-02-18 22:08:13 +02:00
Aliaksandr Valialkin
46e98ed490 vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
54d2f67924 app/vmselect/promql: properly make copies of EvalConfig 2021-02-11 12:41:37 +02:00
Aliaksandr Valialkin
5973fd4067 app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2 
This should prevent from data races if the underlying functions modify EvalConfig contents.
 2021-02-10 23:06:58 +02:00
Aliaksandr Valialkin
985c3e301d app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel 
This should reduce query execution times.
 2021-02-10 23:00:56 +02:00
Aliaksandr Valialkin
8de3a329ff app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax 2021-02-03 20:36:42 +02:00
Aliaksandr Valialkin
7d23f3ff3a app/vmselect: add ability to set an additional label filters via extra_label query arg 2021-02-01 17:42:35 +02:00
Aliaksandr Valialkin
44c74f1e79 app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag 
This should reduce incorrect usage and assumptions for this flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020
 2021-01-22 13:00:35 +02:00
Aliaksandr Valialkin
e55205220b app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time 2021-01-19 22:57:50 +02:00
Aliaksandr Valialkin
c5bdab5a4c app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
 2021-01-13 12:43:47 +02:00
Aliaksandr Valialkin
8cae98aa78 app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
 2021-01-13 10:59:32 +02:00
Aliaksandr Valialkin
df6e399f73 app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d 2021-01-12 16:12:47 +02:00
Aliaksandr Valialkin
47872ada7e app/vmselect/promql: do not ajdust offset value provided in the query 
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
 2020-12-27 14:10:15 +02:00
Aliaksandr Valialkin
0e739efc88 app/vmselect/promql: simplify defer call for querystats.RegisterQuery 2020-12-27 12:07:56 +02:00
Aliaksandr Valialkin
e6deb39064 app/vmselect: refactor /api/v1/stats/top_queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
 2020-12-25 17:24:25 +02:00
Nikolay
76d092c091 Adds query stats handler (#945) 
* Adds query stat handler,
for query and query_range api, victoriametrics tracks query execution time,
stats are expored at /api/v1/status/queries endpoint with topN param
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907

* fixed query stats bugs

* improves queryStats tracker

* improves query stat

* small fix

* fix tests

* added more tests

* fixes 386 tests

* naming fixes

* adds drop for outdated records
 2020-12-25 17:24:24 +02:00
Aliaksandr Valialkin
262cf81757 app/vmselect: properly parse negative combined offsets such as -1h2m3s 
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
 2020-12-19 01:25:03 +02:00
Aliaksandr Valialkin
8d1031c29a app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100 2020-12-15 14:10:50 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
bd8eef2528 app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect 2020-12-08 15:17:05 +02:00
Aliaksandr Valialkin
d242c2f2bd app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N 2020-12-05 12:31:01 +02:00
Aliaksandr Valialkin
8cf76d8747 app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936
 2020-12-03 21:46:18 +02:00
Aliaksandr Valialkin
11bbb3552d app/vmselect/promql: make fmt 2020-12-02 21:34:15 +02:00
Aliaksandr Valialkin
9e98a8f3d3 app/vmselect/promql: return nan from minute(m) when m equals to nan 
This aligns VictoriaMetrics behaviour with Prometheus behaviour.

The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 20:16:40 +02:00
Aliaksandr Valialkin
def513355e app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. 
This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.
 2020-12-02 13:12:33 +02:00
Aliaksandr Valialkin
490c70a958 app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true 
This aligns MetricsQL behavior to Prometheus' one.

The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 12:09:40 +02:00
Aliaksandr Valialkin
4ef7158e89 app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does 2020-12-02 00:28:56 +02:00
Aliaksandr Valialkin
e72ccc9239 app/vmselect: add remoteAddr to slow query log in order to improve debuggability 
This will simplify identifying the client that sends slow queries to VictoriaMetrics.
 2020-11-18 20:40:02 +02:00
Aliaksandr Valialkin
7d76fdedcc app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place 
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
 2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin
2ac5f00d98 app/vmselect: propagate errors from vmstorage to response to the client if -search.denyPartialResponse command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891

This commit also adds `"isPartial":{true|false}` field to `/api/v1/*` responses. `"isPartial":true` is set when the response
is based on a partial data because some of vmstorage nodes weren't available during query processing.
 2020-11-14 13:20:10 +02:00
Aliaksandr Valialkin
8f42e59e05 app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points 
Do not take into account spare data point value if the next point will is located too far from the current point.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 15:23:37 +02:00
Aliaksandr Valialkin
da6d82a8dd app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() 
This should remove unexpected spikes at the end of gaps.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 14:59:32 +02:00
Aliaksandr Valialkin
4f2c5877db app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters 2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin
8f3339fa81 app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin
6385432611 app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum 2020-11-10 01:18:59 +02:00
Aliaksandr Valialkin
efebc3b6fb app/vmselect/promql: code cleanup after 43823addea 2020-11-06 01:31:33 +02:00
n4mine
3127aa92b5 app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883) 2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin
66de02fbb4 app/vmselect/promql: allow dropping trailing sample only for default_rollup function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
 2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin
6b623eba02 app/vmselect/promql: go fmt 2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin
7c0b658865 app/vmselect/promql: do not drop trailing datapoints for instant queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-11-02 00:12:53 +02:00
Aliaksandr Valialkin
c4464594b7 app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K 2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin
ee2902ddaf app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series 
Now trailing data points are additionally dropped for time series with a single raw sample

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-17 10:44:26 +03:00
Aliaksandr Valialkin
b9a4601c97 app/vmselect/promql: return a single time series at max from absent() function like Prometheus does 2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin
217c192c88 app/vmselect/promql: improve time series staleness detection 
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin
f877e703c8 app/vmselect/promql: fix mode_over_time calculations 
Previously `mode_over_time` could return garbage due to improper shuffling of input data points.
 2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin
2d03d0e2dd app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning 
Functions are:

* keep_last_value
* keep_next_value
* interpolate
* running_min
* running_max
* running_avg
* range_min
* range_max
* range_avg
* range_first
* range_last
* range_quantile
* smooth_exponential

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
3881c84afe Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" 
This reverts commit ac45082216.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric names:

* ceil
* floor
* round

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
79d70480b7 Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" 
This reverts commit bb61a4769b.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* clamp_min
* clamp_max

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
8c37b63ea9 Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" 
This reverts commit e5202a4eae.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* max_over_time
* min_over_time
* avg_over_time
* quantile_over_time
* geomean_over_time
* mode_over_time
* holt_winters
* predict_linear

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
f6ee6efc34 app/vmselect/promql: add missing label filters to binary operands before query execution 
This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization

See also https://github.com/cortexproject/cortex/issues/3253
 2020-10-07 21:17:11 +03:00
Aliaksandr Valialkin
3e0c473cc9 app/vmselect/promql: fill gaps on graphs for range_* and running_* functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806
 2020-10-02 14:02:47 +03:00
Aliaksandr Valialkin
db14f22fc0 app/vmselect: stop /api/v1/export/* execution if client disconnects 2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin
b8bce348c5 app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
 2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin
b776a93608 app/vmselect/promql: support composite durations like Prometheus 2.21 does 
The following durations are supported now: `1h5m35s` or `1s543ms`

See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
 2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin
d3ad0d365e app/vmselect: move Deadline from netstorage to searchutils 
This removes dependency on netstorage from searchutils.
 2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin
579c20756a app/vmselect: substitute inf values at smooth_exponential with the previous values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin
d67e6d3d2e app/vmselect: skip infinite values when calculating smooth_exponential 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 11:57:53 +03:00
Aliaksandr Valialkin
f3a79abfb4 app/vmselect/promql: go fmt 2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin
4f06eed1c1 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin
0d0b606455 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin
db91045348 app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3 2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin
804304c365 app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true 2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin
478d8f8393 app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions 
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
 2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin
3490160fd0 app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does 2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin
285665e93b app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3) 2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin
95a8c492ef app/vmselect/promql: properly handle -n^m like Prometheus does 
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
 2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin
7f93d61a56 app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin
01000505a0 app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin
75bff1a567 app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus 
Rollup functions:

  - avg_over_time
  - min_over_time
  - max_over_time
  - quantile_over_time

This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin
8835004a4c app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does 
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin
a04f4a3d9a app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration 2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin
bdb881c43b app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...) 2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin
a2aa3a60eb app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
0c00fe70cf app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
 2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin
338ee47d60 app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window 
Just divide the data point value by the window duration in this case.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
717c554fb0 app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now 
See e00cfc854d for the approach used now.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
d9037b3970 app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
0f6f0d30d3 app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query 2020-07-28 15:14:40 +03:00
Aliaksandr Valialkin
9dccedc599 app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp 
This aligns `group()` behaviour to Prometheus
 2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin
b191e425b3 app/vmselect/promql: improve further the accuracy of buckets_limit() function 
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
 2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin
43871e79c6 app/vmselect/promql: avoid dropping inf bucket in buckets_limit 
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
 2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin
978c1e930e app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets 2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin
51cbf27077 app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function 
Now it properly merges the bucket with the previous bucket after deletion.
 2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin
cf69b1ea6f app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k 
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
 2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin
45334f61de app/vmselect: fix tests for rate_over_sum 2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin
3526e8768a app/vmselect/promql: typo fix after 3e557c9861 2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin
8d1721d128 app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration 
Something like `sum_over_time(m[d]) / d`, but more accurate.
 2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin
88e8bed0c9 app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time 
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
 2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin
5bb4fe1ba4 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
0755cb3b50 app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin
71eba8dcf5 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin
4d2011a87d app/vmselect/promql: add mode() aggregate function 2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin
427fa43ce2 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin
eb402a17bd app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin
ea8dc85ba8 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin
fc8fe38a82 app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin
c64914a7e4 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin
d449d0a0e1 app/vmselect/promql: add missing tests for ifnot binary operation 2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin
7e706eea13 app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation 2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin
6c1a47b5e0 app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function 2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin
fb86071552 app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528
 2020-07-08 19:09:31 +03:00
DexterZhang
9930ce1fa9
Feat/query list vmselect (#575) 
* feat(vmselect): add support for listing current running queries and canceling specific query

* fix(vmselect): change current queries' pid from int64 counter to uuid

* feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth

* fix(vmselect): add more info to current queries

* review: delete some unnecessary code and use function instead of init

* review: returen *queriesMap in newQueriesMap

* review: delete unused var in struct queriesMap, add comments to exported functions

* review: add return if error occurs

* feat(vmselect): truncate query string in current running query list API since the size of query string might be large;
                use query string's pointer in struct `query` for the same reason;
		add query info API to get full access of query's info;
 2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin
f10e8809c0 app/vmselect: add interpolate function for filling gaps with linearly interpolated values 
See https://stackoverflow.com/q/62565021/274937 for details
 2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
nicbaz
46c5c0772c vmselect: fix label_replace when mismatch (#579) 
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".

Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.

Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
 2020-06-23 23:54:29 +03:00
Aliaksandr Valialkin
70bf8218bb app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin
6939e36fdd app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
 2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin
d83c68ca03 app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions 
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
 2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin
8ff28f5b91 app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3 2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin
7d46dd452a app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries 2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin
37068064dd app/vmselect/promql: fix outilersk calculations 2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin
fc81ea38d4 app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series 2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
f3d9a5b0ec app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint 2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin
18a0caee43 app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one 2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin
81b8811cf4 app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag 
Limit the estimated time series count after aggregation with grouping by the number of source time series.
 2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin
408ade27a9 app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y 2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin
21c2982ac8 app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation 2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin
f341c6fcc4 Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors" 
This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c.

Reason for revert: this commit has been removed from single-node version
 2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin
9ed4951ec8 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin
cd1145e5f4 app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors 2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin
b59f1f1504 app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin
603d4c9217 app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( 
Leave both flags for backwards compatibility reasons.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin
a873b553cf app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
 2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin
a012f6fe70 app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions 2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin
58cb7fc476 app/vmselect: adjust label_map() handling for corner cases 
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series

All the unmatched labels are kept unchanged.
 2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin
0e7a71a245 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin
50555d89d3 app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model 2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin
0d893eff36 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
6ca1e58d98 app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick 2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin
b58e3fc8a9 app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-23 18:06:26 +02:00
Yaroslav
c69d4b01f0 fix rollupOpen(), rollupHigh(), rollupLow() functions (#328) 2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin
f22aefdb16 app/vmselect/promql: log when rollupResult cache is cleared 2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin
d5c2a0ce64 app/vmselect: add -search.cacheTimestampOffset command-line flag 
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.

This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin
846d7fa7e9 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
e6d9ea3094 app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8 2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin
4a1de7fee9 app/vmselect/promql: properly adjust time range for data to select 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin
8e77b54846 app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent 2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin
ce38b176bc app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin
4f7116d1ee app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin
45bc6c62f2 app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus 2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin
170c1c3a4e app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
b28c9a3944 app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value 
Examples for such counters: OS-level counters for network or cpu stats.
 2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin
3e304890a6 app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
 2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin
ea53a21b02 all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
e127173984 app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
40e564eb9c app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
ce0b602405 app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args 2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin
bcd3f0c5bd app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin
fc01b11ddc app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval 2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin
16fb128bbc app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions 
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
 2020-01-11 00:26:11 +02:00
Aliaksandr Valialkin
adc36d00b7 app/vmselect/promql: properly handle aggr(aggr_over_time(...)) 2020-01-10 21:57:11 +02:00
Aliaksandr Valialkin
87a106702b app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function 
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
 2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin
c314d9a219 app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions 
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
 2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin
7c6df1e51d app/vmselect/promql: skip rate calculation for the first point on time series 2020-01-08 14:42:44 +02:00
Aliaksandr Valialkin
76707b2ab9 app/vmselect/promql: fix calculations for histogram_share 2020-01-04 14:44:15 +02:00
Aliaksandr Valialkin
accad01b3e app/vmselect/promql: add missing MetricName into netstorage.Result in tests 2020-01-04 12:53:14 +02:00
Aliaksandr Valialkin
6f29d37cb5 app/vmselect/promql: add histogram_share(le, buckets) function 2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin
2290503140 app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 
See https://github.com/prometheus/prometheus/issues/2882
 2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin
67f94bbe12 app/vmselect/promql: add histogram_over_time(m[d]) rollup function 2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin
9a1f6848ca app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time 
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
 2020-01-03 20:44:54 +02:00
Aliaksandr Valialkin
3d0c7b095a app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series 
This should provide better estimation for `rate` in the beginning of time series.
 2020-01-03 19:03:32 +02:00
Aliaksandr Valialkin
6ea7f23446 app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series 2020-01-03 01:02:30 +02:00
Aliaksandr Valialkin
e0abf45d45 app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations 2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin
eb1a66c577 lib/metricsq: add ExpandWithExprs 2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin
453d71d082 Rename lib/promql to lib/metricsql and apply small fixes 2019-12-25 22:09:09 +02:00
Mike Poindexter
009d1559db Split Extended PromQL parsing to a separate library 2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin
ff18101d30 app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range 
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:

    BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
 2019-12-24 22:45:49 +02:00
Aliaksandr Valialkin
e24ee43109 app/vmselect/promql: adjust calculations for rate and increase for the first value 
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
 2019-12-24 19:41:03 +02:00
Aliaksandr Valialkin
9a2554691c app/vmselect/promql: properly calculate rate on the first data point 
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
 2019-12-24 15:55:15 +02:00
Aliaksandr Valialkin
b238997a84 all: rename Extended PromQL to PromQL extensions 2019-12-12 19:29:59 +02:00
Aliaksandr Valialkin
c25b97829f app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed 
Updates https://github.com/prometheus/prometheus/issues/5706
 2019-12-11 14:00:18 +02:00
Aliaksandr Valialkin
f79b61e2a1 app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does 2019-12-11 00:57:54 +02:00
Aliaksandr Valialkin
5d2ff573aa app/vmselect/promql: allow negative offsets 
Updates https://github.com/prometheus/prometheus/issues/6282
 2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
d39bba3547 app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range 
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
 2019-12-05 19:27:45 +02:00
Aliaksandr Valialkin
819bb36852 app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals 
This should improve scrape interval estimation for tiem series with gaps.
 2019-12-02 13:43:04 +02:00
Aliaksandr Valialkin
1e2019b1b6 app/vmselect/promql: fix corner case for increase over time series with gaps 
In this case `increase` could return invalid high value for the first point after the gap.
 2019-11-30 01:34:18 +02:00
Aliaksandr Valialkin
28cc4c09b5 app/vmselect/promql: remove zero timeseries from prometheus_buckets output 2019-11-25 19:10:13 +02:00
Aliaksandr Valialkin
f7da9b2db2 app/vmselect/promql: allow nested parens 2019-11-25 16:13:33 +02:00
Aliaksandr Valialkin
d2619d6dce vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1 2019-11-25 15:22:50 +02:00
Aliaksandr Valialkin
f46fb6c740 app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph 
This should reduce the amounts memory allocations
 2019-11-25 14:24:30 +02:00
Aliaksandr Valialkin
0f184affa7 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
dbd07041ae app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0 2019-11-25 13:44:08 +02:00
Aliaksandr Valialkin
8bb254d960 app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series 2019-11-24 00:04:15 +02:00
Aliaksandr Valialkin
414259f47b app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets 2019-11-23 14:19:19 +02:00
Aliaksandr Valialkin
193d553f6d app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets 2019-11-23 14:02:25 +02:00
Aliaksandr Valialkin
4d76977745 app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile 2019-11-23 11:49:16 +02:00
Aliaksandr Valialkin
5f6f03c692 app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets 2019-11-23 00:21:56 +02:00