Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-22 08:26:19 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6,028 Commits 114 Branches 748 Tags 649 MiB
8bc5e83872
Commit Graph

488 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
5bb4fe1ba4 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
0755cb3b50 app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin
71eba8dcf5 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin
4d2011a87d app/vmselect/promql: add mode() aggregate function 2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin
427fa43ce2 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin
eb402a17bd app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin
ea8dc85ba8 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin
fc8fe38a82 app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin
c64914a7e4 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin
d449d0a0e1 app/vmselect/promql: add missing tests for ifnot binary operation 2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin
7e706eea13 app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation 2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin
6c1a47b5e0 app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function 2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin
fb86071552 app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528
 2020-07-08 19:09:31 +03:00
DexterZhang
9930ce1fa9
Feat/query list vmselect (#575) 
* feat(vmselect): add support for listing current running queries and canceling specific query

* fix(vmselect): change current queries' pid from int64 counter to uuid

* feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth

* fix(vmselect): add more info to current queries

* review: delete some unnecessary code and use function instead of init

* review: returen *queriesMap in newQueriesMap

* review: delete unused var in struct queriesMap, add comments to exported functions

* review: add return if error occurs

* feat(vmselect): truncate query string in current running query list API since the size of query string might be large;
                use query string's pointer in struct `query` for the same reason;
		add query info API to get full access of query's info;
 2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin
f10e8809c0 app/vmselect: add interpolate function for filling gaps with linearly interpolated values 
See https://stackoverflow.com/q/62565021/274937 for details
 2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
nicbaz
46c5c0772c vmselect: fix label_replace when mismatch (#579) 
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".

Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.

Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
 2020-06-23 23:54:29 +03:00
Aliaksandr Valialkin
70bf8218bb app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin
6939e36fdd app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
 2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin
d83c68ca03 app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions 
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
 2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin
8ff28f5b91 app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3 2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin
7d46dd452a app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries 2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin
37068064dd app/vmselect/promql: fix outilersk calculations 2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin
fc81ea38d4 app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series 2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
f3d9a5b0ec app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint 2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin
18a0caee43 app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one 2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin
81b8811cf4 app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag 
Limit the estimated time series count after aggregation with grouping by the number of source time series.
 2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin
408ade27a9 app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y 2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin
21c2982ac8 app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation 2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin
f341c6fcc4 Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors" 
This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c.

Reason for revert: this commit has been removed from single-node version
 2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin
9ed4951ec8 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin
cd1145e5f4 app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors 2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin
b59f1f1504 app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin
603d4c9217 app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( 
Leave both flags for backwards compatibility reasons.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin
a873b553cf app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
 2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin
a012f6fe70 app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions 2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin
58cb7fc476 app/vmselect: adjust label_map() handling for corner cases 
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series

All the unmatched labels are kept unchanged.
 2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin
0e7a71a245 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin
50555d89d3 app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model 2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin
0d893eff36 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
6ca1e58d98 app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick 2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin
b58e3fc8a9 app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-23 18:06:26 +02:00
Yaroslav
c69d4b01f0 fix rollupOpen(), rollupHigh(), rollupLow() functions (#328) 2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin
f22aefdb16 app/vmselect/promql: log when rollupResult cache is cleared 2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin
d5c2a0ce64 app/vmselect: add -search.cacheTimestampOffset command-line flag 
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.

This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin
846d7fa7e9 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
e6d9ea3094 app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8 2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin
4a1de7fee9 app/vmselect/promql: properly adjust time range for data to select 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin
8e77b54846 app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent 2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin
ce38b176bc app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin
4f7116d1ee app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin
45bc6c62f2 app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus 2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin
170c1c3a4e app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
b28c9a3944 app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value 
Examples for such counters: OS-level counters for network or cpu stats.
 2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin
3e304890a6 app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
 2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin
ea53a21b02 all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
e127173984 app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
40e564eb9c app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
ce0b602405 app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args 2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin
bcd3f0c5bd app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin
fc01b11ddc app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval 2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin
16fb128bbc app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions 
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
 2020-01-11 00:26:11 +02:00
Aliaksandr Valialkin
adc36d00b7 app/vmselect/promql: properly handle aggr(aggr_over_time(...)) 2020-01-10 21:57:11 +02:00
Aliaksandr Valialkin
87a106702b app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function 
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
 2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin
c314d9a219 app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions 
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
 2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin
7c6df1e51d app/vmselect/promql: skip rate calculation for the first point on time series 2020-01-08 14:42:44 +02:00
Aliaksandr Valialkin
76707b2ab9 app/vmselect/promql: fix calculations for histogram_share 2020-01-04 14:44:15 +02:00
Aliaksandr Valialkin
accad01b3e app/vmselect/promql: add missing MetricName into netstorage.Result in tests 2020-01-04 12:53:14 +02:00
Aliaksandr Valialkin
6f29d37cb5 app/vmselect/promql: add histogram_share(le, buckets) function 2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin
2290503140 app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 
See https://github.com/prometheus/prometheus/issues/2882
 2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin
67f94bbe12 app/vmselect/promql: add histogram_over_time(m[d]) rollup function 2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin
9a1f6848ca app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time 
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
 2020-01-03 20:44:54 +02:00
Aliaksandr Valialkin
3d0c7b095a app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series 
This should provide better estimation for `rate` in the beginning of time series.
 2020-01-03 19:03:32 +02:00
Aliaksandr Valialkin
6ea7f23446 app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series 2020-01-03 01:02:30 +02:00
Aliaksandr Valialkin
e0abf45d45 app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations 2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin
eb1a66c577 lib/metricsq: add ExpandWithExprs 2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin
453d71d082 Rename lib/promql to lib/metricsql and apply small fixes 2019-12-25 22:09:09 +02:00
Mike Poindexter
009d1559db Split Extended PromQL parsing to a separate library 2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin
ff18101d30 app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range 
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:

    BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
 2019-12-24 22:45:49 +02:00
Aliaksandr Valialkin
e24ee43109 app/vmselect/promql: adjust calculations for rate and increase for the first value 
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
 2019-12-24 19:41:03 +02:00
Aliaksandr Valialkin
9a2554691c app/vmselect/promql: properly calculate rate on the first data point 
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
 2019-12-24 15:55:15 +02:00
Aliaksandr Valialkin
b238997a84 all: rename Extended PromQL to PromQL extensions 2019-12-12 19:29:59 +02:00
Aliaksandr Valialkin
c25b97829f app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed 
Updates https://github.com/prometheus/prometheus/issues/5706
 2019-12-11 14:00:18 +02:00
Aliaksandr Valialkin
f79b61e2a1 app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does 2019-12-11 00:57:54 +02:00
Aliaksandr Valialkin
5d2ff573aa app/vmselect/promql: allow negative offsets 
Updates https://github.com/prometheus/prometheus/issues/6282
 2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
d39bba3547 app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range 
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
 2019-12-05 19:27:45 +02:00
Aliaksandr Valialkin
819bb36852 app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals 
This should improve scrape interval estimation for tiem series with gaps.
 2019-12-02 13:43:04 +02:00
Aliaksandr Valialkin
1e2019b1b6 app/vmselect/promql: fix corner case for increase over time series with gaps 
In this case `increase` could return invalid high value for the first point after the gap.
 2019-11-30 01:34:18 +02:00
Aliaksandr Valialkin
28cc4c09b5 app/vmselect/promql: remove zero timeseries from prometheus_buckets output 2019-11-25 19:10:13 +02:00
Aliaksandr Valialkin
f7da9b2db2 app/vmselect/promql: allow nested parens 2019-11-25 16:13:33 +02:00
Aliaksandr Valialkin
d2619d6dce vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1 2019-11-25 15:22:50 +02:00
Aliaksandr Valialkin
f46fb6c740 app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph 
This should reduce the amounts memory allocations
 2019-11-25 14:24:30 +02:00
Aliaksandr Valialkin
0f184affa7 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
dbd07041ae app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0 2019-11-25 13:44:08 +02:00
Aliaksandr Valialkin
8bb254d960 app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series 2019-11-24 00:04:15 +02:00
Aliaksandr Valialkin
414259f47b app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets 2019-11-23 14:19:19 +02:00
Aliaksandr Valialkin
193d553f6d app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets 2019-11-23 14:02:25 +02:00
Aliaksandr Valialkin
4d76977745 app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile 2019-11-23 11:49:16 +02:00
Aliaksandr Valialkin
5f6f03c692 app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets 2019-11-23 00:21:56 +02:00
Aliaksandr Valialkin
77bb66a5be app/vmselect/promql: properly calculate integrate(q[d]) 2019-11-13 21:11:03 +02:00
Aliaksandr Valialkin
c33640664a app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
 2019-11-13 20:26:07 +02:00
Aliaksandr Valialkin
33abbec6b4 app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions 
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.

Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.

This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
 2019-11-08 19:37:43 +02:00
Aliaksandr Valialkin
4a8251feff app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q 2019-11-01 12:21:43 +02:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
99786c2864 app/vmselect/prometheus: add -search.maxLookback command-line flag for overriding dynamic calculations for max lookback interval 
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.

The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
 2019-10-15 21:37:17 +03:00
Aliaksandr Valialkin
a5302a6651 app/vmselect/promql: take into account the previous point when calculating max_over_time and min_over_time 
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
 2019-10-08 12:30:16 +03:00
Aliaksandr Valialkin
ba460f62e6 app/vmselect/promql: do not generate timestamps for NaN values in timestamp function according to Prometheus logic 2019-09-27 18:55:16 +03:00
Aliaksandr Valialkin
bd1cf053f6 app/vmselect/promql: add increases_over_time and decreases_over_time functions 
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
 2019-09-25 20:38:51 +03:00
Aliaksandr Valialkin
ee4585db33 app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step])) 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
 2019-09-13 21:42:11 +03:00
Aliaksandr Valialkin
828e5f6d26 app/vmselect/promql: binary operation fixes according to Prometheus behaviour 
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
 2019-09-13 17:43:09 +03:00
Aliaksandr Valialkin
d0953e9f02 app/vmselect/promql: ignore grouping by destination label in count_values, since such a grouping is performed automatically 2019-09-04 19:59:02 +03:00
Aliaksandr Valialkin
f78ffe565f app/vmselect/promql: do not return artificial points beyond the last point in time series 2019-09-04 16:34:29 +03:00
Aliaksandr Valialkin
b08f085082 app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does 2019-09-03 20:43:29 +03:00
Aliaksandr Valialkin
604a4312f9 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
e9db22a551 app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
 2019-08-22 14:39:24 +03:00
Aliaksandr Valialkin
90a4b00b10 app/vmselect/promql: fix panic on -search.disableCache 
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
 2019-08-21 17:12:01 +03:00
Aliaksandr Valialkin
491b1762c8 app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue 2019-08-21 11:29:41 +03:00
Aliaksandr Valialkin
99331606e1 app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries 
This should reduce memory allocations for big number of timeseries
 2019-08-20 23:01:57 +03:00
Aliaksandr Valialkin
1101765adb app/vmselect/promql: add label_value(q, label_name) func, which returns numeric value labels with name label_name in q 2019-08-20 00:28:44 +03:00
Aliaksandr Valialkin
940349ccb9 app/vmselect/promql: independently track offset hints for tStart and tEnd 
This should improve performance if timeseries starts or ends on the selected time range
 2019-08-19 13:40:24 +03:00
Aliaksandr Valialkin
6ae4b4190f app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do 
This should improve the performance of queries over big number of time series
with big number of output points.
 2019-08-19 13:03:38 +03:00
Aliaksandr Valialkin
005aabd305 app/vmselect/promql: add scrape_interval(q[d]) function, which would return scrape interval for q over d 2019-08-18 21:08:15 +03:00
Aliaksandr Valialkin
218cb4623a app/vmselect/promql: hande comparisons with NaN similar to Prometheus 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
 2019-08-18 00:25:58 +03:00
Aliaksandr Valialkin
dcce92c63c app/vmselect/promql: add lifetime(q[d]) function, which returns the lifetime of q over d in seconds. 
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
 2019-08-16 11:59:51 +03:00
Aliaksandr Valialkin
0cb66a8f95 app/vmselect/promql: fix corner-case calculation for ideriv 2019-08-16 11:59:50 +03:00
Aliaksandr Valialkin
1b5b9ced27 app/vmselect/promql: properly handle corner cases for rollup functions 2019-08-15 23:31:28 +03:00
Aliaksandr Valialkin
b8bbe92de1 app/vmselect/promql: store compressed results in the cache 
This should increase rollup results cache capacity.
 2019-08-14 02:32:16 +03:00
Aliaksandr Valialkin
8c2158af24 all: use workingsetcache instead of fastcache 
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.

The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
 2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin
880b1d80b1 app/vmselect: optimize /api/v1/series by skipping storage data 
Fetch and process only time series metainfo.
 2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
000c154641 app/vmselect/promql: tune automatic window adjustement 
Increase the windows adjustement for small scrape intervals,
since they usually have higher jitter.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
 2019-08-04 19:34:11 +03:00
Aliaksandr Valialkin
1d4ddadbb1 app/vmselect/promql: further increase the allowed jitter for scrape interval 
Real-world production data shows higher jitter than 1/8 of scrape interval.
This may results in gaps on the graph. So increase the allowed jitter to 1/4
of scrape interval in order to reduce the probability of gaps on the graphs
over time series with high jitter for scrape_interval.
 2019-08-02 20:16:41 +03:00
Aliaksandr Valialkin
ade7bc30db app/vmselect/promql: tolerate higher jitter in scrape interval 
Allow jitter for up to 1/8 instead of 1/16 for the scrape interval.
This should imrpove graphs when `step` is smaller than the `scrape_interval`.
 2019-08-01 23:25:53 +03:00
Aliaksandr Valialkin
c994fbf500 app/vmselect/promql: add vm_slow_queries_total metric for counting slow queries 
The query is slow if its execution time exceeds `-search.logSlowQueryDuration`
 2019-07-31 03:36:45 +03:00
Aliaksandr Valialkin
071a122119 app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken 2019-07-31 01:18:11 +03:00
Aliaksandr Valialkin
b9a16b93e7 app/vmselect/promql: allow adjusting window for default rollup function 
Default rollup function is `last_over_time`. It must support adjusting
the provided window in order to prevent from gaps on the graph
for window values smaller than scrape interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
 2019-07-31 00:45:58 +03:00
Aliaksandr Valialkin
c901a6472f app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
 2019-07-30 22:05:55 +03:00
Aliaksandr Valialkin
aac482517f app/vmselect/promql: return NaN from count() over zero time series 
This aligns `count` behavior with Prometheus.
 2019-07-25 22:02:34 +03:00
Aliaksandr Valialkin
0e52357f35 app/vmselect/promql: properly calculate incremental aggregations grouped by __name__ 
Previously the following query may fail on multiple distinct metric names match:

    sum(count_over_time{__name__!=''}) by (__name__)
 2019-07-25 21:53:26 +03:00
Aliaksandr Valialkin
092c9b39a8 app/vmselect/promql: remove empty time series after applying filters like q > 0 
This should reduce CPU and RAM usage for queries over high number of time series.
 2019-07-12 19:59:49 +03:00
Aliaksandr Valialkin
6875fb411a app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores 
This may reduce response times for aggregation over big number of time series
with small step between output data points.
 2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin
cbab86fd9d app/vmselect/promql: reduce RAM usage for aggregates over big number of time series 
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
 2019-07-10 13:03:36 +03:00
Aliaksandr Valialkin
ba8195c58e all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
df6f17b82c app/vmselect/promql: mention -search.logSlowQueryDuration flag value in the slow query log message 2019-07-10 00:43:01 +03:00
Aliaksandr Valialkin
73ae889244 app/vmselect/promql: extract rmoeveGroupTags function for removing unneeded tags from MetricName according to the given modifierExpr 2019-07-09 23:20:58 +03:00
Aliaksandr Valialkin
603b34edbd app/vmselect/promql: properly preserve metric name after applying functions in any case from transformFuncsKeepMetricGroup 2019-07-09 23:10:49 +03:00
Aliaksandr Valialkin
bba07d05fe app/vmselect/promql: remove empty timeseries left after topk call 2019-07-04 19:43:07 +03:00
Aliaksandr Valialkin
858746fa6c app/vmselect/promql: gracefully handle duplicate timestamps in irate and rollup_rate funcs 
Previously such timestamps result in `+Inf` results. Now the previous timestamp is used
for the calculations.
 2019-07-03 12:41:30 +03:00
Aliaksandr Valialkin
a3abed80ff app/vmselect: do not return empty time series in /api/v1/query result 2019-07-01 17:16:26 +03:00
Aliaksandr Valialkin
c3c60bee45 app/vmselect: add -search.denyPartialResponse flag for disabling partial responses if some of vmstorage nodes are unavailable 
Also accept `deny_partial_response` query arg in Prometheus API handlers. If it is set to true,
then return error if some of vmstorage nodes are unavailable.
 2019-06-30 01:27:07 +03:00
Aliaksandr Valialkin
72a3050c41 app/vmselect/promql: consistency renaming: candlestick -> rollup_candlestick 2019-06-29 03:13:25 +03:00
Aliaksandr Valialkin
96342f1422 app/vmselect: add candlestick(m[d]) func for returning open, close, low and high rollups on the given time range d 
This function is frequently used in financial apps. See https://en.wikipedia.org/wiki/Candlestick_chart
 2019-06-27 18:46:54 +03:00
Aliaksandr Valialkin
4f54bcf90b app/vmselect/promql: suppress error when template func is used inside modifier list. Just leave it as is 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/78
 2019-06-25 20:43:57 +03:00
Aliaksandr Valialkin
d0bf4393a9 app/vmselect/promql: increase default value for -search.maxPointsPerTimeSeries from 10k to 30k 
This may be required for subqueries with small steps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/77
 2019-06-24 22:53:25 +03:00
Aliaksandr Valialkin
334cf253c7 app/vmselect/promql: adjust value returned by linearRegression to the end of time range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
 2019-06-24 22:46:03 +03:00
Aliaksandr Valialkin
14cd628948 app/vmselect/promql: add sum2 and sum2_over_time, geomean and geomean_over_time funcs. 
These functions may be useful for statistic calculations.
 2019-06-24 16:45:00 +03:00
Aliaksandr Valialkin
0eac538fc8 app/vmselect/promql: adjust the provided window only for range functions with dt in denominator 
This should fix range function calculations such as `changes(m[d])` where `d` is smaller
than the scrape interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
 2019-06-23 19:27:25 +03:00
Aliaksandr Valialkin
ec57e59154 app/vmselect/promql: use deriv_fast instead of deriv in ttf, since deriv calculations have been changed recently 2019-06-23 15:54:12 +03:00
Aliaksandr Valialkin
516062b162 app/vmselect/promql: adjust ttf calculation, so deriv(freev) for freev=m[d] could be properly calculated 2019-06-23 14:31:36 +03:00
Aliaksandr Valialkin
a4e040f5ef app/vmselect/promql: typo fixes in comments 2019-06-21 23:22:54 +03:00
Aliaksandr Valialkin
c05d443791 app/vmselect/promql: add deriv_fast function for calculating fast derivative 
`deriv_fast` calculates derivative based on the first and the last point on the interval
instead of calculating linear regression based on all the data points on the interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
 2019-06-21 23:05:48 +03:00
Aliaksandr Valialkin
98eafdbd58 app/vmselect/promql: use linear regression in deriv func like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
 2019-06-21 22:54:34 +03:00
Aliaksandr Valialkin
f334908c22 app/vmselect/promql: ajdust data model to the model used in Prometheus 
Do not take into account data points on the range `[timestamp .. timestamp+step)`
when calculating value on the given `timestamp`.
Use only data points from the past when performing these calculations like Prometheus does.

This should reduce discrepancies between results returned by VictoriaMetrics
and results returned by Prometheus.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
 2019-06-21 21:55:25 +03:00
Aliaksandr Valialkin
837e349b7d app/vmselect/promql: do not strip __name__ form time series after binary comparison operation 
Example:

  foo > 10

Would leave `foo` name for all the matching time series on the left.
 2019-06-21 13:08:02 +03:00
Aliaksandr Valialkin
e84b7641ef app/vmselect/promql: remove unused func keepLastValue; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:35:19 +03:00
Aliaksandr Valialkin
db042bf6d6 app/vmselect/promql: typo fix; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:33:52 +03:00
Aliaksandr Valialkin
947bc16f8c app/vmselect/promql: use dynamic limit on memory for concurrent queries 2019-06-12 23:18:23 +03:00
Aliaksandr Valialkin
8567e3463d app/vmselect/promql: merge non-overlapping duplicate time series in group_left and group_right joins 2019-06-12 20:33:01 +03:00
Aliaksandr Valialkin
88005237f4 app/vmselect/promql: swap binary operation with modifier in the error message for improved readability 2019-06-12 17:14:33 +03:00
Aliaksandr Valialkin
a71381ad2a app/vmselect/promql: list a sample of duplicate time series in the error message for group_left or group_right 
This should improve troubleshooting for complex queries involving `group_left` and `group_right` modifiers.
 2019-06-12 16:57:34 +03:00
Aliaksandr Valialkin
eea7da8e0c app/vmselect/promql: prevent from count_values explosion of timeseries, which could result in OOM 2019-06-11 01:03:18 +03:00
Aliaksandr Valialkin
e87a602209 app/vmselect/promql: skip superflouos timestamps copying in count_values 2019-06-11 00:44:09 +03:00
Aliaksandr Valialkin
ec84febc1c app/vmselect/promql: remove superflouos timeseries copy in histogram_quantile func 2019-06-11 00:39:35 +03:00
Aliaksandr Valialkin
1fab34fb5c app/vmselect/promql: remove superflouos timeseries copy in union func 2019-06-11 00:35:09 +03:00
Aliaksandr Valialkin
a6f368499d app/vmselect/promql: skip NaN values in count_values func 2019-06-10 22:42:41 +03:00
Aliaksandr Valialkin
913f888d0c app/vmselect/promql: properly handle {__name__ op "string"} queries 
This has been broken in 7294ef333ad26f4f6578b783e97649e58b1f8945 .
 2019-06-07 02:02:09 +03:00
Aliaksandr Valialkin
8f4790625d app/vmselect/promql: return the correct time series from quantile 
Previously arbitrary time series could be returned from `quantile`
depending on sort order for the last data point in the selected range.

Fix this by returning the calculated time series.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/55
 2019-06-06 17:33:53 +03:00
Aliaksandr Valialkin
2ff0d595b0 app/vmselect/promql: add -search.disableCache flag for disabling response caching 
This may be useful for data back-filling, when the response caching
could interfere badly with newly added data points with timestamps
in the past.
 2019-06-04 17:30:41 +03:00
Aliaksandr Valialkin
53242105fb app/vmselect/promql: allow escaping identifiers with \ and \xXX 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/42
 2019-05-31 17:35:54 +03:00
Aliaksandr Valialkin
a2c71f18a3 app/vmselect/promql: misspeling fix 2019-05-25 21:53:48 +03:00
Aliaksandr Valialkin
bdf696ef18 all: fix misspellings 2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
121a920a18 Makefile: add -s flag to go fmt in make fmt command 2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin
2ff996e276 app/vmselect: log slow queries if their execution time exceeds -search.logSlowQueryDuration 2019-05-24 16:14:46 +03:00
Aliaksandr Valialkin
8e3eb5b39d app/vmselect/promql: add alias(q, name) function that sets the given name to all the time series in q 2019-05-24 02:42:10 +03:00
Aliaksandr Valialkin
bb048937bc app/vmselect/promql: add label_transform(q, label, regexp, replacement) function for replacing all the occurences of regexp with replacement in the given label for q 2019-05-23 16:26:07 +03:00
Aliaksandr Valialkin
24578b4bb1 all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00