Aliaksandr Valialkin
b1ded7cf9a
app/vmselect/promql: add absent_over_time(m[d])
func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0
app/vmselect/promql: add histogram_over_time(m[d])
rollup function
2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
3e09d38f29
app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time
...
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
a774120460
app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate
for the first point on the time series
...
This should provide better estimation for `rate` in the beginning of time series.
2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin
b5645ccbdf
app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something)
from 100 to 1K, since 100 may result in OOM for high number of time series
2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin
cb3a342882
app/vmselect/promql: add share_le_over_time
and share_gt_over_time
functions for SLI and SLO calculations
2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
6f67e0b56b
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43
Split Extended PromQL parsing to a separate library
2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
d880325cf6
app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range
...
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:
BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin
9e19949c6b
app/vmselect/promql: adjust calculations for rate
and increase
for the first value
...
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin
0455c03cb9
app/vmselect/promql: properly calculate rate
on the first data point
...
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin
5b75984aa9
app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs
2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin
bea5a8700a
app/vmselect: add -search.maxExportDuration
command-line flag for limiting /api/v1/export
duration
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275
2019-12-20 11:35:22 +02:00
Aliaksandr Valialkin
96ff8d9adb
app/vmselect: add ability to pass match[]
, start
and end
to /api/v1/labels
...
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.
See https://github.com/prometheus/prometheus/issues/6178 for more details.
2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin
cea5a14853
all: rename Extended PromQL
to PromQL extensions
2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin
73b2a3d4b7
app/vmselect/promql: return lower
and upper
bounds for the estimated percentile from histogram_quantile
if third arg is passed
...
Updates https://github.com/prometheus/prometheus/issues/5706
2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin
caa4eb72d9
app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query
like Prometheus does
2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin
3c076544bf
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin
c09472dfd9
app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median}
aggregate functions for returning the exact k time series on the given time range
...
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin
face3d57bf
app/vmselect: add placeholders for /api/v1/rules
and /api/v1/alerts
2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin
20812008a7
lib/storage: remove metricID with missing metricID->metricName entry
...
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.
Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
9476b73527
app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals
...
This should improve scrape interval estimation for tiem series with gaps.
2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin
75ff524a4e
app/vmselect/promql: fix corner case for increase
over time series with gaps
...
In this case `increase` could return invalid high value for the first point after the gap.
2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin
4c859d980c
app/vmselect/prometheus: consistently apply nocache
arg to /api/v1/query
the same way ast to /api/v1/query_range
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin
110235f789
app/vmselect/prometheus: fix content-type for /api/v1/export
responses
...
The correct Content-Type should be `application/stream+json` instead of `application/json`
Thanks to Joshua Ryder for pointing to this.
2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin
205233d9a7
app/vmselect/promql: remove zero timeseries from prometheus_buckets
output
2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin
3f99f39e9b
app/vmselect/prometheus: reduce default value for -search.latencyOffset
from 60s to 30s
...
30 seconds should be enough for almost all the cases
2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin
e91cb34c0e
app/vmselect/promql: allow nested parens
2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin
826dfd63a5
vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1
2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin
0401969d78
app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph
...
This should reduce the amounts memory allocations
2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin
da98703748
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
c28876172f
app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0
2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin
50ae1879c6
app/vmselect/promql: add histogram
aggregate function, which is useful for building heatmaps from multiple time series
2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin
8582b50360
app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets
2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin
19dfe52254
app/vmselect/promql: properly handle histogram_quantile(0, ...)
with zero buckets
2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin
4bb88843cf
app/vmselect: add vm_per_query_{rows,series}_processed_count
histograms
2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
7753c8c0a1
app/vmselect/promql: transparently apply prometheus_buckets
in histogram_quantile
2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin
c4287b3c86
app/vmselect/promql: add prometheus_buckets
function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics
to Prometheus-compatible buckets
2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin
1f3fd2c910
app/vmselect: adjust end
arg instead of adjusting start
arg if start > end
...
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin
7a4635f853
all: remove the remaining mentions of cluster version
2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
0094bc4fc9
app/vmselect/prometheus: properly adjust too big time time
on /api/v1/query
...
Too big `time` must be adjusted to `now()-queryOffset`.
2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin
3f1637fae8
app/vmselect/promql: properly calculate integrate(q[d])
2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin
3fd32e331a
app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
2019-11-13 20:24:26 +02:00
Oleg Kovalov
b4f44befa3
fix misspelled words ( #229 )
2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
63b05c0b9f
app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions
...
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.
Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.
This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
2f58f37f07
app/vmselect/promql: add lag(q[d])
function, which returns the lag between the current timestamp and the timstamp for the last data point in q
2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin
b0295dbf2e
app/vmselect: add -search.latencyOffset
flag for tuning the time after data collection when data points become visible in query results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin
97ce4e03a5
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
c66a691593
app/vmselect/prometheus: add -search.maxLookback
command-line flag for overriding dynamic calculations for max lookback interval
...
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.
The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
956fdd89d3
app/vmselect/promql: take into account the previous point when calculating max_over_time
and min_over_time
...
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
2019-10-08 12:30:05 +03:00
Aliaksandr Valialkin
4e9e1ca0f7
app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially
...
This became the case after b7ee2e7af2
.
2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin
c1d3705be0
app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock
...
This allows re-using the destination buffer for marshaling in the outer loop.
2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin
b7ee2e7af2
app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series
2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin
67d44b0845
app/vmselect/promql: do not generate timestamps for NaN values in timestamp
function according to Prometheus logic
2019-09-27 18:54:43 +03:00
Aliaksandr Valialkin
26dc21cf64
app/vmselect/promql: add increases_over_time
and decreases_over_time
functions
...
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin
d4218d27e6
app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step]))
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-13 21:41:04 +03:00
Aliaksandr Valialkin
4d636c244d
app/vmselect/promql: binary operation fixes according to Prometheus behaviour
...
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
2019-09-13 17:42:52 +03:00
Aliaksandr Valialkin
6954e126fc
app/vmselect/promql: ignore grouping by destination label in count_values
, since such a grouping is performed automatically
2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin
305e7bc981
app/vmselect/promql: do not return artificial points beyond the last point in time series
2019-09-04 16:35:34 +03:00
Aliaksandr Valialkin
9721d06c6a
app/vmselect/prometheus: do not adjust start
and end
args in /api/v1/query_range
if nocache=1
arg is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171
2019-09-04 13:10:09 +03:00
Aliaksandr Valialkin
db4560ca31
app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does
2019-09-03 20:42:54 +03:00
Aliaksandr Valialkin
1575a560f0
app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks
...
The maximum inmemory file size now depends on `-memory.allowedPercent`.
This should improve performance and reduce the number of filesystem calls
on machines with big amounts of RAM when performing heavy queries
over big number of samples and time series.
2019-09-03 13:32:09 +03:00
Aliaksandr Valialkin
9196c085a7
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
603a79b357
app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries
2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin
ac004665b5
all: return 503 http error if service is temporarily unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin
1272e407b2
app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
2019-08-22 14:39:46 +03:00
Aliaksandr Valialkin
f4a38f7fb1
app/vmselect/promql: fix panic on -search.disableCache
...
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin
ab740afd0d
app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue
2019-08-21 11:29:24 +03:00
Aliaksandr Valialkin
7b5168adfb
app/vmselect/promql: remove NaNs from /api/v1/query_range
output like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
2019-08-20 23:01:41 +03:00
Aliaksandr Valialkin
a0d480fbf3
app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries
...
This should reduce memory allocations for big number of timeseries
2019-08-20 23:01:39 +03:00
Aliaksandr Valialkin
d3fd113a80
app/vmselect/promql: add label_value(q, label_name)
func, which returns numeric value labels with name label_name
in q
2019-08-20 00:28:34 +03:00
Aliaksandr Valialkin
dd86e6130c
app/vmselect/promql: independently track offset hints for tStart and tEnd
...
This should improve performance if timeseries starts or ends on the selected time range
2019-08-19 13:40:14 +03:00
Aliaksandr Valialkin
6a27657d73
app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do
...
This should improve the performance of queries over big number of time series
with big number of output points.
2019-08-19 13:03:29 +03:00
Aliaksandr Valialkin
e74fb23189
app/vmselect/promql: add scrape_interval(q[d])
function, which would return scrape interval for q
over d
2019-08-18 21:08:26 +03:00
Aliaksandr Valialkin
582fdc059a
app/vmselect/promql: hande comparisons with NaN
similar to Prometheus
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-08-18 00:25:50 +03:00
Aliaksandr Valialkin
1c108fc494
app/vmselect/promql: add lifetime(q[d])
function, which returns the lifetime of q
over d
in seconds.
...
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
2019-08-16 11:59:32 +03:00
Aliaksandr Valialkin
d6b5ed6d39
app/vmselect/promql: fix corner-case calculation for ideriv
2019-08-16 11:59:28 +03:00
Aliaksandr Valialkin
639b14e8ab
app/vmselect/promql: properly handle corner cases for rollup functions
2019-08-15 23:29:59 +03:00
Aliaksandr Valialkin
9ecb994671
app/vmselect/promql: store compressed results in the cache
...
This should increase rollup results cache capacity.
2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin
09fc6e22e5
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin
5c83f8e203
app: add vm_concurrent_
metrics for visibility in concurrency limiters for vminsert and vmselect
2019-08-05 18:30:57 +03:00
Aliaksandr Valialkin
99e048c9df
app/vmselect: allow passing match[]
, start
and time
to /api/v1/label/<label_name>/values
...
`/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)`
call in Grafana templating.
2019-08-04 23:09:21 +03:00
Aliaksandr Valialkin
47e4b50112
app/vmselect: optimize /api/v1/series
by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin
241170dc05
app/vmselect/prometheus: prevent from fetching and scanning all the data on /api/v1/searies
call by default
2019-08-04 19:42:36 +03:00
Aliaksandr Valialkin
1c69f4eadc
app/vmselect/promql: tune automatic window adjustement
...
Increase the windows adjustement for small scrape intervals,
since they usually have higher jitter.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
2019-08-04 19:34:05 +03:00
Aliaksandr Valialkin
8d93b15b86
app/vmselect/promql: further increase the allowed jitter for scrape interval
...
Real-world production data shows higher jitter than 1/8 of scrape interval.
This may results in gaps on the graph. So increase the allowed jitter to 1/4
of scrape interval in order to reduce the probability of gaps on the graphs
over time series with high jitter for scrape_interval.
2019-08-02 20:10:23 +03:00
Aliaksandr Valialkin
f090b2e917
app/vmselect/promql: tolerate higher jitter in scrape interval
...
Allow jitter for up to 1/8 instead of 1/16 for the scrape interval.
This should imrpove graphs when `step` is smaller than the `scrape_interval`.
2019-08-01 23:26:00 +03:00
Aliaksandr Valialkin
3a38b23fa3
app/vmselect/promql: add vm_slow_queries_total
metric for counting slow queries
...
The query is slow if its execution time exceeds `-search.logSlowQueryDuration`
2019-07-31 03:36:37 +03:00
Aliaksandr Valialkin
8bd6f1f6df
app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken
2019-07-31 01:18:07 +03:00
Aliaksandr Valialkin
4aaa5c2efc
app/vmselect/promql: allow adjusting window for default rollup function
...
Default rollup function is `last_over_time`. It must support adjusting
the provided window in order to prevent from gaps on the graph
for window values smaller than scrape interval.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
2019-07-31 00:45:54 +03:00
Aliaksandr Valialkin
10f5a26bec
app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
2019-07-30 22:05:10 +03:00
Aliaksandr Valialkin
afcc7fb167
app/vmselect/netstorage: improve error message when reading data blocks from storage
...
Mention the block number in the error. This should simplify troubleshooting in this code.
2019-07-28 12:12:35 +03:00
Aliaksandr Valialkin
17096b5750
app/vmselect/promql: return NaN from count()
over zero time series
...
This aligns `count` behavior with Prometheus.
2019-07-25 22:02:30 +03:00
Aliaksandr Valialkin
66efa5745f
app/vmselect/promql: properly calculate incremental aggregations grouped by __name__
...
Previously the following query may fail on multiple distinct metric names match:
sum(count_over_time{__name__!=''}) by (__name__)
2019-07-25 21:53:20 +03:00
Aliaksandr Valialkin
6a943a6a58
app/vmselect/promql: remove empty time series after applying filters like q > 0
...
This should reduce CPU and RAM usage for queries over high number of time series.
2019-07-12 19:59:27 +03:00
Aliaksandr Valialkin
ab88890523
app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores
...
This may reduce response times for aggregation over big number of time series
with small step between output data points.
2019-07-12 15:52:22 +03:00
Aliaksandr Valialkin
62a7353479
app/vmselect/prometheus: set start
arg in /api/v1/series
to the minimum allowed time by default as Prometheus does
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
2019-07-11 17:10:14 +03:00
Aliaksandr Valialkin
54bd21eb4a
app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage
2019-07-11 17:07:20 +03:00
Aliaksandr Valialkin
a336bb4e22
app/vmselect/promql: reduce RAM usage for aggregates over big number of time series
...
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
2019-07-10 13:04:39 +03:00