Commit Graph

740 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
0580a58feb
app/vmselect/promql: add support for @ modifier
Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier

Extend the support with the following features:
* Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()`
* Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)`
  returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348
2022-01-13 22:15:19 +02:00
Aliaksandr Valialkin
7d635a2311
app/vmselect/promql: fix limit_offset() test
The test has been broken in addae7fc6a
2022-01-13 16:58:47 +02:00
Yury Molodov
7ee09286ef
vmui: switching to Preact (#2053)
* feat: replace @codemirror to text field

* feat: switch to Preact from React

* fix: optimize mui imports

* fix: remove unused vars

* update package-lock.json

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-01-11 10:34:01 +02:00
Aliaksandr Valialkin
cf5553d18e
app/vmselect/vmui: run make vmui-update after 89ff7b2465 2022-01-03 21:06:34 +02:00
Yury Molodov
85dc41007b
vmui: legend fixes (#1995)
* feat: add a reset query by clicking the logo

* feat: add sequence number for query fields

* feat: invert behavior on the graph's legend

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-23 12:15:21 +02:00
Roman Khavronenko
d1c474c5b3
vmselect: update /query_exemplars placeholder (#2000)
Grafana expects `data` in response to be a slice and logs an err
if it is not:
```
err="[]v1.ExemplarQueryResult: decode slice: expect [ or n, but found , error found in #0 byte of ...||..., bigger context ...||..."
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-12-23 12:08:40 +02:00
Yury Molodov
0952caca24
vmui: add custom start range (#1989)
* feat: add custom start range

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-21 20:22:48 +02:00
Yury Molodov
4c9e563c89
vmui: graph fixes (#1982)
* fix: remove disabling custom step when zooming

* feat: add a dynamic calc of the width of the graph

* fix: add validate y-axis limits

* fix: correct axis limits for value 0

* fix: change logic create time series

* fix: change types for tooltip

* fix: correct points on the line

* fix: change the logic for set graph width

* fix: stop checking the period when auto-refresh is enabled

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-20 17:37:45 +02:00
Aliaksandr Valialkin
b393a16a25
app/vmselect/vmui: make vmui-update 2021-12-20 13:50:55 +02:00
Aliaksandr Valialkin
3396f08874
app/vmselect/promql: follow-up after 177e345d8a
* Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions.
* Simplify their implementation
* Mention these functions in docs/CHANGELOG.md
2021-12-20 13:15:06 +02:00
匠心零度
cc4725843b
add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972)
Co-authored-by: lirenzuo <lirenzuo@shein.com>
2021-12-20 13:15:05 +02:00
Aliaksandr Valialkin
c2b0a6109b
app/vmselect/prometheus: typo fix after bf0e3ad2171219b7883b149999c4dc0ac15b9797 2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin
7c3239966e
app/vmselect/vmui: make vmui-update 2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin
d9f075622b
app/vmselect: de-duplicate data exported via /api/v1/export/csv by default
Previously the exported data wasn't de-duplicated.
Now it is possible to export the raw data without deduplication
by passing reduce_mem_usage=1 query arg to /api/v1/export/csv

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837
2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin
c170841951
app/vmselect/promql: add timestamp_with_name(m[d]) function
This function works the same as `timestamp()`, but doesn't remove source time series names.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388
2021-12-15 23:38:09 +02:00
Yury Molodov
603be150c4
vmui: introduce application mode (#1949)
* feat: add a label for the Query field

* fix: change zoom position

* fix: add description and error code to alerts

* fix: correct logic query history

* fix: correct update query history

* feat: add custom step

* update package-lock.json

* feat: introduce application mode

* build vmui

* Revert "build vmui"

This reverts commit c0e2415550.

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-15 21:34:04 +02:00
Aliaksandr Valialkin
bc3923111b
lib/storage: return dedup interval in milliseconds from GetDedupInterval()
This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.
2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin
cdfe854c9b
lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge()
This improves the code readability and debuggability, since the output of these functions
stops depending on global state.
2021-12-14 20:52:29 +02:00
Yury Molodov
b3389ce26b
vmui: custom step (#1942)
* feat: add a label for the Query field

* fix: change zoom position

* fix: add description and error code to alerts

* fix: correct logic query history

* fix: correct update query history

* feat: add custom step

* update package-lock.json

* docs: document that VMUI now supports overriding of `step` query arg, which is passed to `/api/v1/query_range`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-14 14:52:44 +02:00
Yury Molodov
28655e8eb8
vmui: minor fixes (#1936)
* feat: add a label for the Query field

* fix: change zoom position

* fix: add description and error code to alerts

* fix: correct logic query history

* fix: correct update query history

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document bugfixes

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-13 13:44:32 +02:00
Aliaksandr Valialkin
3d86e6353b
app/vmselect/vmui: make vmui-update 2021-12-13 13:33:35 +02:00
Yury Molodov
6367edae1d
vmui: add a label for the Query field (#1923)
* feat: add a label for the Query field

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-12 19:10:56 +02:00
Aliaksandr Valialkin
addae7fc6a
app/vmselect/promql: preserve the order of time series passed to limit_offset() function
Previously time series passed to `limit_offset()` were shuffled according to hash for their labels.
This was unexpected behaviour for most users.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951
2021-12-12 18:07:15 +02:00
Yury Molodov
39e29a50f1
vmui: multiple queries (#1916)
* feat: change duration by "enter"

* fix: optimize data processing for chart

* feat: set minimum step to 1ms

* update dependencies

* feat: remove save the last query to local storage

* fix: handle an error in a table with subqueries

* feat: store display type in URL

* Revert "feat: store display type in URL"

This reverts commit ccc242c69a.

* feat: store display type in URL

* refactor: move the time setting to a folder

* refactor: move the query configurator to a folder

* refactor: move the auth settings to a folder

* feat: improve styles

* feat: add multi query

* update package-lock

* feat: add display multiple queries

* feat: add limits for multiple queries

* update dependencies

* feat: add history for multiple queries

* feat: add line type to legend

* feat: change style for switch

* feat: change the logic for axes limits for multiple queries

* update package-lock.json

* update dependencies

* feat: add the filter to legend

* wip

* lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':'

This allows copy-pasting the url to http server from logs.

* lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':'

This allows copy-pasting the url to http server from logs.

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-08 16:43:18 +02:00
Aliaksandr Valialkin
54d4a1c959
app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863
2021-12-06 17:33:49 +02:00
Aliaksandr Valialkin
df810cfc9a
app/vmselect/vmui: make vmui-update 2021-12-06 10:18:03 +02:00
Aliaksandr Valialkin
120fedf1ac
app/vmselect/vmui: make vmui-update 2021-11-30 01:38:59 +02:00
Yury Molodov
500945499f
vmui: optimize render (#1852)
* feat: change duration by "enter"

* fix: optimize data processing for chart

* feat: set minimum step to 1ms

* update dependencies

* update package-lock

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-11-30 00:57:35 +02:00
Aliaksandr Valialkin
648d788a45
app/vmselect/vmui: make vmui-update 2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin
d5ae9e9619
app/vmselect/vmui: make vmui-update 2021-11-29 12:23:12 +02:00
Michael Fuller
8a94dd36d1
vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838)
Co-authored-by: Michael Fuller <mfuller@digitalocean.com>
2021-11-29 12:14:55 +02:00
Aliaksandr Valialkin
84decc4254
app/vmselect/promql: consistently return zero from deriv(const) 2021-11-17 18:01:58 +02:00
Aliaksandr Valialkin
aa6d11a33f
app/vmselect/promql: add now() function, which returns the current timestamp as a floating-point value in seconds 2021-11-17 16:42:11 +02:00
Aliaksandr Valialkin
1eb756fd11
app/vmselect/promql: arrange function names in the code in alphabetical order
This should simplify code maintenance in the future
2021-11-14 13:54:43 +02:00
Aliaksandr Valialkin
82cf9f26db
app/vmselect/promql: prevent from incorrect calculations for deriv() over multiple samples with identical timestamps 2021-11-12 13:53:30 +02:00
Aliaksandr Valialkin
4fb19fe34b
all: consistently return application/json content-type without charset=utf-8
The `application/json` content-type has utf-8 encoding by default.
See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin
31486f2244
app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
2021-11-09 11:42:43 +02:00
Yury Molodov
89e0f66bfc
vmui: migration MUI Core v4 to v5 (#1795)
* migration MUI Core v4 to v5

* app/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-11-08 15:46:57 +02:00
Aliaksandr Valialkin
3a1ac218ba
app/vmselect/promql: add duration_over_time(m[d], max_interval) function
This function calculates the actual lifetime of the time series on the given lookbehdind window `d`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780
2021-11-08 13:14:57 +02:00
Yury Molodov
654686fd40
vmui: fix graph reset (#1788)
* feat: add query history

* fix: change detect keyUp for nav query history

* feat: set default query history

* feat: change graph legend

* update dependencies

* update codemirror version

* fix: correct update period time after zoom/pan

* fix: optimize data processing for the graph

* fix: eliminate memory leaks related to mouse events

* fix: correct display of straight line

* Merge branch 'master' into vmui-fix-reset-graph

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-11-05 20:58:49 +02:00
Aliaksandr Valialkin
be9c7713d7
app/vmselect: make vmui-update 2021-11-04 12:16:55 +02:00
Aliaksandr Valialkin
a102fca4ac
app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778
2021-11-03 16:03:10 +02:00
Aliaksandr Valialkin
43bf0333eb
app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names 2021-11-03 13:20:34 +02:00
Yury Molodov
a16c88095c
vmui: correct migration material-ui (#1758)
* migration material-ui

* fix: rollback popover

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-27 20:43:24 +03:00
Aliaksandr Valialkin
99784b21c1
all: fix build issues and tests for Apple M1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653
2021-10-27 15:07:19 +03:00
Yury Molodov
e7591577f0
vmui: update dependencies (#1754)
* update dependencies

* update codemirror version

* app/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-26 19:32:31 +03:00
Aliaksandr Valialkin
d8ce86d685
app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738
2021-10-24 13:32:25 +03:00
Yury Molodov
3ccc37a98b
vmui: query history (#1732)
* feat: add query history

* fix: change detect keyUp for nav query history

* feat: set default query history

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-22 12:29:27 +03:00
Yury Molodov
744f6a98eb
vmui: features (#1711)
* feat: initial uPlot graph

* feat: add zoom/pan for graph

* fix: add zoom by ctrl/mac

* fix: remove unused code

* feat: add toggle cache for fetch

* feat: add fix y-axis limits

* fix: stop point events while panning

* fix: change getting cursor position when scaling

* feat: add cursor tooltip to graph

* fix: uninstall chart.js

* fix: change link for create an issue

* fix: set default cache value to true

* app/vmalert: follow-up after 0e2486df56

* docs/CHANGELOG.md: document 5416e18007

* app/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-18 15:19:01 +03:00
Aliaksandr Valialkin
ed994873fd
app/vmselect/promql: randomize the static selection of time series returned from limitk()
Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series.
Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.
2021-10-16 21:16:34 +03:00
Aliaksandr Valialkin
da0adf446d
app/vmselect/promql: typo fix in comment: didsn't -> didn't 2021-10-16 13:00:10 +03:00
Aliaksandr Valialkin
48c3668b3d
app/vmselect/promql: add atan2 binary operator, which is going to be added in Prometheus 2.31
See https://github.com/prometheus/prometheus/pull/9248
2021-10-11 21:17:23 +03:00
Aliaksandr Valialkin
b3d28f3872
app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31
See https://github.com/prometheus/prometheus/issues/9233
2021-10-11 21:17:21 +03:00
Aliaksandr Valialkin
f1e1d20ac4
app/vmselect/promql: consistently return the same set of time series from limitk() function
This is the expected behaviour by most users.
2021-10-08 19:55:29 +03:00
Aliaksandr Valialkin
cf87050d83
app/vmui: follow-up after 7bfb44113e
* Run `vmui-update`
* Document the changes in README.md and CHANGELOG.md
2021-10-08 15:11:09 +03:00
Aliaksandr Valialkin
cbf01f7384
app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow
The list of functions, which can adjust lookbehind window is more limited than the rest of functions,
so it is better from maintainability and readability PoV using the allowlist instead of blocklist.
2021-10-07 13:19:22 +03:00
Aliaksandr Valialkin
7e6ec0bbe9
app/vmselect/promql: return back the behaviour for deriv() function when the lookbehind window doesnt contain enough points
It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d`
contains less than 2 samples in the same way as `rate()` does.

This is a follow-up after 3e084be06b .
2021-10-07 12:55:58 +03:00
Roman Khavronenko
0a8804d6aa
app/vmselect: make predict_linear and deriv compatible with Prometheus (#1681)
Previously, `predict_linear` returned slightly different results comparing
to Prometheus. The change makes linear regression algorithm compatible
with Prometheus.

`deriv` was excluded from the list of functions which can adjust the time
window for the same reasons.
2021-10-07 12:55:56 +03:00
Aliaksandr Valialkin
ec6eb03d65
app/vmselect/promql: add topk_last and bottomk_last functions 2021-09-30 13:23:27 +03:00
Roman Khavronenko
95ddfda894
app/vmselect: fix binary comparison func (#1667)
The fix makes the binary comparison func to check for NaNs
before executing the actual comparison. This prevents VM
to return values for non-existing samples for expressions
which contain bool comparisons. Please see added test
for example.
2021-09-30 13:04:10 +03:00
Roman Khavronenko
57eb5ef194
app/vmselect: fix testRowsEqual func NaN checks (#1666)
It appeared, that `testRowsEqual` NaN comparison was incorrect.
The fix caused some tests to fail. Please see the change and
tests updated.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-30 13:04:08 +03:00
Yury Molodov
a6037a8ce8
vmui: fixed bug with time range (time zone) (#1661)
* fix: set date in query string in utc format

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-29 01:01:27 +03:00
Roman Khavronenko
444a8882f4
app/vmselect: disable time-window adjustment for min/max_over_time funcs (#1658)
Adjustment results into discrepancy between Prometheus and VM on time windows
smaller than scrape interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:43:54 +03:00
Roman Khavronenko
f46f847531
app/vmselect: always return zero for stddev func if there is only one value (#1659)
The fix will always return zero if received set of items consists of one
element only, which also means no deviation.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:42:19 +03:00
Roman Khavronenko
72468c9d75
app/vmselect: return NaN instead of 0 for empty value sets (#1660)
The change affects `count/stddev/stdvar_over_time` funcs and makes
them to return NaN instead of zero when there is no datapoints
in a time window.
This is needed for improving compatibility with Prometheus.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:38:24 +03:00
Roman Khavronenko
c8ac8f44b2
app/vmselect: rm quantile_over_time fast-path optimisations (#1662)
The removed fast path optimisations weren't consistent with
`quantile` function behavior and results into discrepancy.
Specifically, results didn't match in cases when:
* 0 < phi > 1;
* values contain only one element.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:35:59 +03:00
Yury Molodov
80df31b2ee
vmui: use Chart.js as default engine for graph (#1634)
* feat: add Plotly as default engine for graph

* fix: remove unused components

* feat: use Chart.js as default engine graph

* fix: correct styles for loader

* feat: add zoom/pan for chart

* feat: add height for chart

* fix: remove unused code

* fix: remove empty units from duration

* fix: change debounce for pan to 500ms

* fix: add utility for plugins register globally

* fix: optimize render graph

* feat: add buffer data for zoom

* fix: add limits for zoom in/out

* fix: change update data while zooming

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-27 22:27:49 +03:00
Aliaksandr Valialkin
4b3951fd86
app/vmselect/promql: add rollup_scrape_interval(m[d]) function
It calculates the min, max and avg scrape intervals for m over the given lookbehind window d
2021-09-27 19:21:55 +03:00
Aliaksandr Valialkin
eff31c10ec
app/vmselect/promql: follow-up after 526dd93b32
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
2021-09-27 18:57:03 +03:00
Roman Khavronenko
6061464d80
app/vmselect: quantile func compatiblity with Prometheus (#1646)
* app/vmselect: `quantile` func compatiblity with Prometheus

The `quantile` func was previously calculated by https://github.com/valyala/histogram
package. The result of such calculation was always the closest real value to
requested quantile. While in Prometheus implementation interpolation is used.
Such difference may result into discrepancy in output between Prometheus and
VictoriaMetrics.

This commit adds a Prometheus-like `quantile` function. It also used by other
functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: `quantile` review fixes

* quantile functions were split into multiple to provide
different API for already sorted data;
* float64sPool is used for reducing allocations. Items in pool may have
different sizes, but defining a new pool was complicates due to name collisions;

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-27 18:57:02 +03:00
Aliaksandr Valialkin
a746fe8778
app/vmselect/promql: follow-up after 57b3320478 2021-09-24 01:25:04 +03:00
Roman Khavronenko
7ac0559d4f
app/vmselect: make sorting for query result similar to Prometheus (#1647)
* app/vmselect: make sorting for query result similar to Prometheus

Updated sorting allows to get the order of series in result similar or equal
to what Prometheus returns.
The change is needed for compatibility reasons.

* Update app/vmselect/promql/exec_test.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-24 01:25:03 +03:00
Aliaksandr Valialkin
1115c2f235
app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus
Previously VictoriaMetrics could return unexpected result of the right-hand side operand
had multiple time series with the given set of labels mentioned in `on(labels)`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643
2021-09-24 00:56:35 +03:00
Aliaksandr Valialkin
3f1104ea74
app/vmselect: fix accessing /graphite/* endpoints 2021-09-21 14:06:18 +03:00
n4mine
a5fcc9e356
fix: typo, dddresses -> addresses (#1630) 2021-09-20 14:27:34 +03:00
Aliaksandr Valialkin
76c650bbda app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once 2021-09-17 23:35:04 +03:00
Aliaksandr Valialkin
81d24045c2 app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function
This function calculates multiple quantiles over the given buckets at once

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
2021-09-17 13:39:06 +03:00
Aliaksandr Valialkin
4ed7548ced app/vmselect/promql: optimize quantiles() calculation
Calculate quantiles in one go instead of calculating each quantile individually
2021-09-17 12:38:21 +03:00
f41gh7
348f705d95 follow-up after e2f99214f3e518f79c1c96ff918820a69d350ba1 2021-09-16 13:56:28 +03:00
Roman Khavronenko
05bf6cca82 vmselect: add stub for graphite/functions requests
The request `graphite/functions` is sent by Grafana datasource
and results into error logs in vmselect.
2021-09-16 13:51:37 +03:00
Aliaksandr Valialkin
3f8de4f82f app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL 2021-09-16 13:35:38 +03:00
Aliaksandr Valialkin
a4eaf1a0a1 app/vmselect/prometheus: make more clear log messages for errors during sending data to remote clients 2021-09-16 13:35:37 +03:00
Aliaksandr Valialkin
7ad54041fe app/{vminsert,vmselect}: automatically add missing port in -storageNode lists passed to vminsert and vmselect
This should simplify manual setup of the cluster according to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup
2021-09-15 18:04:30 +03:00
Aliaksandr Valialkin
794d941125 app/vmui: follow-up after 777ff75874
The commit contains the following changes:

- Show vmui when requesting /graph page in order to be compatible with Prometheus datasource in Grafana.
- Properly encode query args at vmui url.
- Set the number of points on the graph to the number of horizontal pixels divided by 2. Previously it was hardcoded to 30.
- Do not save server url to persistent storage at browser, since it should be always obtained from the url.
- Run `make vmui-update` for updating vmui embedded into VictoriaMetrics.
2021-09-15 17:41:00 +03:00
Aliaksandr Valialkin
19516aedb5 app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function 2021-09-13 13:48:18 +03:00
Aliaksandr Valialkin
9b07cb2988 app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600
2021-09-09 19:24:10 +03:00
Aliaksandr Valialkin
3788d4f4eb app/vmselect: show useful endpoints when requested /select/<accountID>/ page 2021-08-29 12:05:14 +03:00
Aliaksandr Valialkin
39bb6bdd79 app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
2021-08-27 18:40:25 +03:00
Aliaksandr Valialkin
335de30083 app/vmselect/promql: make fmt after 0078486ea7 2021-08-23 23:05:34 +03:00
Aliaksandr Valialkin
40b06e84f8 app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus
See https://github.com/prometheus/prometheus/pull/8457 for details.
2021-08-23 11:46:29 +03:00
Aliaksandr Valialkin
91534057a3 app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d])
This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d
2021-08-21 19:56:56 +03:00
Roman Khavronenko
1ccb77904b vmselect: update vm_request_duration_seconds value when request fails (#1537)
Before, metric `vm_request_duration_seconds` was update only on successful
attempts which could be misleading. For example, timeout errors on netstorage
request may be not accounted in the metric and won't be visible on dashboards.
Using `defer` statement to update the metric after query arguments validation
may improve the situation.
2021-08-19 14:07:00 +03:00
Aliaksandr Valialkin
ee1f3414d1 app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval
Since most users do not expect the overriding of explicitly set `[d]`.
2021-08-19 10:33:10 +03:00
Aliaksandr Valialkin
5d92fafc40 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin
49886ecbc8 app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541
2021-08-17 13:22:15 +03:00
Aliaksandr Valialkin
38065bec7b app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability 2021-08-17 11:00:58 +03:00
Aliaksandr Valialkin
5420c3d967 app/vmselect/promql: drop staleness marks before calling rollupConfig.Do
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2021-08-15 13:22:26 +03:00
Aliaksandr Valialkin
6c4c54eaad Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions"
This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346.

It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc
2021-08-15 13:22:24 +03:00
Aliaksandr Valialkin
af4a306d7b app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions
Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2021-08-14 12:45:31 +03:00
Aliaksandr Valialkin
c1f81f08d4 all: add support for Prometheus staleness markers
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin
b35ae791f1 app/vmselect: make vmui-update after the commit 4ae14df864a7e327955f44941295a286175423b3 2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin
60cfa5f100 app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
2021-08-03 12:21:53 +03:00
Nikolay
3f3ad13753 adds /rules and /alerts api for grafana (#1504)
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-08-02 17:29:49 +03:00
Aliaksandr Valialkin
99004a6a40 app/vmselect/netstorage: unpack time series data in mostly local big chunks
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
2021-07-30 12:26:33 +03:00
Aliaksandr Valialkin
cbb81c2ce9 app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17 2021-07-30 10:28:19 +03:00
Aliaksandr Valialkin
b709fa387a app/vmselect: follow-up for ed95bc9531
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493
2021-07-29 09:48:47 +03:00
arnoldyahad
ed95bc9531
Add case prometheus/rules for grafana 8 (#1502) 2021-07-29 06:15:35 +03:00
Aliaksandr Valialkin
49bf3abf67 app/vmselect: follow-up for 626073bca8
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
2021-07-28 18:00:04 +03:00
Aliaksandr Valialkin
5d255846ac all: add go:build lines for Go1.17
See https://tip.golang.org/doc/go1.17#gofmt for more details
2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin
3921d8afae app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel 2021-07-26 15:50:45 +03:00
Aliaksandr Valialkin
c3e6ce1db9 app/vmselect: make vmui-update after a91d41f12a 2021-07-26 10:32:01 +03:00
Aliaksandr Valialkin
b047feeb8b app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin
b92702f6d5 app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores 2021-07-16 00:35:36 +03:00
Aliaksandr Valialkin
df117f85bd app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries
This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries
or queries that take too much cpu time.
2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
5830ce2706 app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
6c42db87a8 app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems 2021-07-15 15:40:53 +03:00
Aliaksandr Valialkin
9add9d86a6 app/vmselect/promql: duration handling improvements in MetricsQL queries
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin
f5fa177141 Revert "app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui"
This reverts commit 885a79def6799f288e14df05b35a12569659ab85.

Reason for revert: Grafana doesn't allows accessing /select/<accountID>/prometheus/vmui :(
2021-07-12 09:08:43 +03:00
Aliaksandr Valialkin
ddaa12050d app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui
The /select/<accountID>/prometheus/vmui is needed for accessing via server-side Prometheus datasource for Grafana.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-10 12:52:25 +03:00
Aliaksandr Valialkin
0b98f6c7ff app/vmselect: expose vmui at /vmselect/<accountID>/vmui/ instead of /vmselect/<accountID>/prometheus/vmui/
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-10 12:32:21 +03:00
Aliaksandr Valialkin
98e049ba6d app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-09 17:13:51 +03:00
Aliaksandr Valialkin
2c5e1cd893 app/vmselect: move web ui from /ui to /select/<accountID>/prometheus/ui
This way the UI is available for every tenant (aka accountID) and the UI can automatically determine the needed per-tenant datasource path from page referer.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-08 13:14:50 +03:00
Aliaksandr Valialkin
acb7a95c64 app/vmselect: follow-up after aa11ef6d3b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4
2021-07-07 17:45:09 +03:00
tony
aa11ef6d3b
add vmui for vmselect component (#1431)
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-07-07 17:04:23 +03:00
Aliaksandr Valialkin
9c19719ad6 app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms 2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin
ceda2b1df4 lib/httpserver: print full requestURI in httpserver.Errorf
This should simplify debugging.
2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin
44855f0c9b app/{vmselect,vmstorage}: clarify the description for -dedup.minScrapeInterval command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1426
2021-07-02 15:06:41 +03:00
Aliaksandr Valialkin
856aecae05 app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect 2021-06-23 14:18:37 +03:00
Aliaksandr Valialkin
b92d110cad app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration 2021-06-18 19:07:03 +03:00
Roman Khavronenko
a15c947045 promql: fix increase_pure calculation for cases with stale series (#1381)
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
2021-06-15 17:37:51 +03:00
John Belmonte
3e79f3994e spelling fix: synonym (#1363) 2021-06-11 10:58:48 +03:00
Aliaksandr Valialkin
e8e7f03394 app/vmselect/promql: typo fix in the comment 2021-06-09 18:34:57 +03:00
Aliaksandr Valialkin
a0b001bfec app/vmselect/netstorage: remove duplicate limiter on concurrent queries
It duplicates the `-search.maxConcurrentRequests` limiter.
2021-05-24 19:13:04 +03:00
Aliaksandr Valialkin
180829b8c2 app/vmselect/promql: add timezone_offset(tz) function
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
2021-05-20 11:54:06 +03:00
Nikolay
be87be34a4 Adds tsdb match filters (#1282)
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin
56b08390f6 app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode
`vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.
2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin
6dc5d3b357 all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com 2021-04-20 20:20:01 +03:00
Aliaksandr Valialkin
e48fd5776b app/vmselect/promql: fix tests after d3fa0ccabd 2021-04-08 00:17:40 +03:00
Aliaksandr Valialkin
f90bf265f4 app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-08 00:10:33 +03:00
Aliaksandr Valialkin
9ce3b7e1dd app/vmselect: return data:null instead of data:[] from /api/v1/query_exemplars, since Grafana throws an error otherwise
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-07 23:34:01 +03:00
Aliaksandr Valialkin
3d5f1f779f app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-07 14:16:02 +03:00
Aliaksandr Valialkin
7fce4e9fb4 app/vmselect: return dumb response on /api/v1/query_exemplars request
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-05 23:28:21 +03:00
Aliaksandr Valialkin
2585058a5f Makefile: prepare arm64 and amd64 release archives for cluster version on make release command 2021-04-05 23:01:45 +03:00
Aliaksandr Valialkin
4aa59cae7c app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label) 2021-04-03 22:06:50 +03:00
Aliaksandr Valialkin
b9469de410 app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value") 2021-04-02 23:56:30 +03:00
Aliaksandr Valialkin
4028d692f5 app: do not process non-GET requests on at / handler 2021-04-02 22:56:38 +03:00
Aliaksandr Valialkin
31d3d02d25 app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
2021-04-02 16:50:02 +03:00
Aliaksandr Valialkin
d8c769e6af app/vmselect: add -search.maxStatusRequestDuration command-line flag for limiting the duration of requests to /api/v1/status/* and /api/v1/series/count 2021-03-30 21:39:32 +03:00
Aliaksandr Valialkin
0e29d48628 app/vmselect: remove mentioning for -search.storageTimeout from command-line flag descriptions
This is a follow-up for ecfd6fe78d
2021-03-30 15:07:40 +03:00
Aliaksandr Valialkin
ecfd6fe78d app/vmselect: remove -search.storageTimeout command-line flag, since it has the same meaning as -search.maxQueryDuration
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2021-03-30 14:54:36 +03:00