1204 Commits

Author	SHA1	Message	Date
Nikolay	48041305a2	app/vmselect: adds proxy for rules API (#2516) ... * app/vmselect: adds proxy for rules and alerts API It allows to visualization for rules at grafana https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739 * Update app/vmselect/main.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-05-04 10:12:15 +03:00
Dima Lazerka	b5fce4190c	Code cleanup (#343) ... * Small code cleanup: remove Request from params * Extract common params to all export handlers * Renamed ExportParams -> exportParams * wip Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-03 15:53:58 +03:00
Aliaksandr Valialkin	4fa3cd701c	docs/Cluster-VictoriaMetrics.md: typo fix: serparated -> separated	2022-05-03 15:20:11 +03:00
Aliaksandr Valialkin	75bd7f1539	app/vmui: fix up/down arrow keys on multi-line query after a186434b50	2022-05-03 10:51:08 +03:00
Aliaksandr Valialkin	361b08c30e	lib/storage: leave the last sample per each discrete interval during the deduplicaton ... This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin	7ca32c21c8	app/vmui: execute query by pressing enter in the same way as Prometheus does ... Multi-line query can be entered via `shift-enter` in the query input field	2022-05-02 20:24:43 +03:00
Yury Molodov	6292185abf	vmui: support node v.18 (#2529) ... * fix: add support vmui node18 * fix: remove @mui/styles (legacy styling solution) * Update app/vmui/Dockerfile-build * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 20:24:42 +03:00
Nikolay	022b2875ad	app/vmselect: adds API /api/v1/status/buildinfo (#2515) ... * app/vmselect: adds API /api/v1/status/buildinfo it should fix an compability error with grafana 8.5 prometheus datasource https://github.com/grafana/grafana/pull/46771 * Update main.go	2022-04-29 13:02:43 +03:00
Aliaksandr Valialkin	90f9ed260e	app/vmselect/vmui: make vmui-update after da04e9d1de	2022-04-29 13:02:43 +03:00
Dima Lazerka	e6ee235707	Export "null" in jsonl instead of NaN (#2518) ... * Export "null" in jsonl instead of NaN The NaN appeared because of staleness markers that were added for compatibility. I think it's better to use json `null`, implemented here. Also maybe it also makes sense to add a flag like `?skip-staleness-markers=true` to `/export`, to skip nulls at all? * Update app/vmselect/prometheus/export.qtpl * app/vmselect/prometheus/export.qtpl.go: `make quicktemplate-gen` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-29 12:51:03 +03:00
Yury Molodov	dac203320a	vmui: expression alias (#2495) ... * feat: add alias for queries * docs: update docs for predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-27 10:59:51 +03:00
Aliaksandr Valialkin	6eb1580158	app/vmselect/promql: add tlast_change_over_time(m[d]) function, which returns the timestamp for the last change of m on the given lookbehind window d	2022-04-27 10:58:40 +03:00
Yury Molodov	eae6f68be2	vmui: add support relative time (#2504) ... * feat: add support relative time * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-26 15:46:47 +03:00
Aliaksandr Valialkin	dac24aa342	app/vmselect/promql: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries ... Previously `vector` time series could be unexpectedly returned from such queries	2022-04-21 15:34:14 +03:00
Aliaksandr Valialkin	bc54ae9608	app/vmselect/promql: fix comparison to nan ... The comparison to nan has been broken in d335cc886c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2022-04-21 14:56:41 +03:00
Aliaksandr Valialkin	a5cfe5d13e	app/vmselect/promql: add drop_common_labels() function	2022-04-21 14:20:36 +03:00
Aliaksandr Valialkin	7ca08945fd	app/vmselect/promql: fix q default b where b may have empty time series	2022-04-21 11:43:09 +03:00
Aliaksandr Valialkin	25b841c6ed	app/vmselect/promql: fix duplicate time series error on joins against time series filtered by values ... This should prevent from `duplicate time series` errors when executing the following query: kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1) where `kube_pod_status_phase{phase=~"Pending|Running"}==1` filters out diplicate time series	2022-04-20 22:21:20 +03:00
Aliaksandr Valialkin	0ef7a05fc0	app/vmselect/promql: rename removeNaNs() to more clear removeEmptySeries()	2022-04-20 19:53:24 +03:00
Yury Molodov	60c4e0022a	fix: prevent graph hiding without data (#2456) ... * fix: prevent graph hiding without data * fix: add yaxis labels default * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 17:16:17 +03:00
Aliaksandr Valialkin	3e2a4c07cd	app/vmui: further improvements for number display on graphs ... This is a follow-up for c4d2cd8336 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2409	2022-04-12 16:02:37 +03:00
Aliaksandr Valialkin	4197c61ef8	app/vmselect: make vmui-update	2022-04-12 14:35:37 +03:00
Aliaksandr Valialkin	3605e1743b	app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests	2022-04-11 12:32:23 +03:00
Aliaksandr Valialkin	a4a15a462b	app/vmselect/netstorage: bump RPC API versions for vmselect->vmstorage communications ... This is a follow-up after b843f0e229	2022-04-08 12:36:04 +03:00
Aliaksandr Valialkin	feff0ef286	app/vmselect: make vmui-update	2022-04-01 12:55:53 +03:00
Yury Molodov	b2f0acaca4	vmui: grid support for predefined panels (#2386) ... * update packages * feat: add setting width for predefined panels * docs: update doc by predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-01 12:49:24 +03:00
Yury Molodov	7cbf19812d	vmui: predefined panels (#2243) ... * feat: add basic components for predefined dashboards * fix: change display alert * feat: add autosize and unit for axes * feat: add component for CircularProgress * feat: change layout for predefined dashboards * feat: add override step for predefined panels * feat: add override step for predefined panels * feat: change yaxis limits for predefined panels * fix: rename flag for hide legend * feat: add formatted panel description * feat: add README.md for dashboard setup * feat: validate dashboard settings * feat: add unit for y-ticks * fix: correct display error for dashboards * fix: disable auto refresh after route change * update package-lock.json * fix: add basename for BrowserRouter * fix: add dynamic basename for routing * update packages * feat: add a pre-defined dashboard "per-job resource usage" * feat: display unit in the hover-tooltip * fix: change routing and home layout * fix: change axis width calc * updated packages * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 13:05:21 +02:00
Aliaksandr Valialkin	b843f0e229	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:28:14 +02:00
Aliaksandr Valialkin	84b3234f3d	docs/CHANGELOG.md: document a1e17e91f8	2022-03-21 15:37:03 +02:00
Aliaksandr Valialkin	85695faa94	app/vmselect/bufferedwriter: suppress trivial network errors, which can be generated by remote side ... These errors include `broken pipe` and `reset by peer`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2334	2022-03-18 19:27:33 +02:00
Aliaksandr Valialkin	95da5e52c4	app/vmselect/vmui: make vmui-update	2022-03-18 13:18:57 +02:00
Yury Molodov	235ef947ed	vmui: update packages (#2264) ... * update package.json * update package-lock.json * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-03 18:50:44 +02:00
Aliaksandr Valialkin	99ee1c13c0	app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K ... There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues	2022-02-24 04:03:28 +02:00
Aliaksandr Valialkin	244c23ea2c	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:42:27 +02:00
Aliaksandr Valialkin	701eb0495b	app/vmselect: make vmui-update	2022-02-22 21:02:20 +02:00
Aliaksandr Valialkin	89ead3daca	app/vmselect/netstorage: report vmstorage errors to vmselect clients even if partial responses are allowed ... If a vmstorage is reachable and returns an application-level error to vmselect, then such error must be returned to the caller even if partial responses are allowed, since it usually means cluster mis-configuration. Partial responses may be returned only if some vmstorage nodes are temporarily unavailable. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1941 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/678	2022-02-21 21:17:05 +02:00
Yury Molodov	cdd632985f	vmui: change logic autocomplete (#2196) ... * fix: change autocomplete display logic * fix: change z-index for labels of input fields * fix: change autocomplete display logic * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:44:54 +02:00
Aliaksandr Valialkin	c723ac6a8f	app/vmui: small fixes ... * Remove unneeded dependency on `numeral` package * Properly parse numbers obtained from /api/v1/query_range according to https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats * Optimize updating processing the received data from /api/v1/query_range * Make smoother zoom on `ctrl+scroll` * Reduce the number of points received from /api/v1/query_range by 2x in order to reduce load on backend	2022-02-14 16:31:31 +02:00
Aliaksandr Valialkin	989668beba	app/vmselect/promql: return at most one time series from absent_over_time() in the same way as Prometheus does ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130	2022-02-12 15:46:43 +02:00
Roman Khavronenko	791cad8c2e	lib/promscrape: support prometheus-like duration in scrape configs (#2169) ... * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:51 +02:00
Aliaksandr Valialkin	d1c17fe385	app/vmselect/vmui: make vmui-update	2022-02-08 17:29:35 +02:00
Aliaksandr Valialkin	1173964d8d	app/vmselect/promql: clarify comments on why the right side of if and and operators are executed at first	2022-02-03 00:26:42 +02:00
Aliaksandr Valialkin	10367d7e4c	app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values ... Such filters may slow down time series search, so just skip them. This is a follow-up for e7f1ceeb84 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827	2022-02-02 23:42:25 +02:00
Aliaksandr Valialkin	5f266370c5	all: follow-up after 4bdd10ab90 ... Properly use new bytesutil.Resize* functions	2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin	13c2692ca1	app/vmselect/promql: properly handle foo or bar queries ... Such queries may miss `bar` results after the commit e7f1ceeb84 because common label filters from `foo` could be mistakenly applied to `bar`.	2022-02-01 17:43:10 +02:00
Aliaksandr Valialkin	02b2bfcff3	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations ... Follow-up for f4989edd96 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	784cdcb080	app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first ... PromQL and MetricsQL operators are case-insensitive	2022-01-31 22:07:35 +02:00
Aliaksandr Valialkin	04d6596298	app/vmselect/promql: optimize queries, which join on _info metrics. ... Automatically add common filters from one side of binary operation to the other side before sending the query to storage subsystem. See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/ and https://www.robustperception.io/exposing-the-software-version-to-prometheus	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	d95d945046	app/vmselect/vmui: make vmui-update	2022-01-31 12:42:21 +02:00
Aliaksandr Valialkin	a3eafd2e7f	app/vmselect/vmui: make vmui-update after e3995572bb	2022-01-25 17:52:37 +02:00
Aliaksandr Valialkin	6232eaa938	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions ... Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin	b5abea0e36	app/vmselect/vmui: make vmui-update	2022-01-24 12:25:00 +02:00
Yury Molodov	196bef8348	vmui: fixed display type switching (#2088) ... * fix: correct switch display type * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 16:57:14 +02:00
Yury Molodov	034012c80f	vmui: fix time range selector (#2085) ... * fix: add date validate for time range * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 12:03:41 +02:00
Aliaksandr Valialkin	e8900c90a8	app/vmui: fix last 6 months time range picker ... Previously it was incorrectly selecting 6 minutes instead of 6 months. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960	2022-01-18 23:28:49 +02:00
Aliaksandr Valialkin	f4a9ae28f3	app/vmui: properly calculate graph range for y axis ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037	2022-01-18 22:23:40 +02:00
Aliaksandr Valialkin	f31776d0e4	app/vmui: reduce the refresh interval during graph scrolling/zooming from 1 second to 300 milliseconds ... One second feels too laggy, so let's reduce the refresh interval to 300 milliseconds. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064	2022-01-18 22:23:40 +02:00
Yury Molodov	1d19303f35	fix: remove buffer period (#2078) ... * fix: remove buffer period * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the implemented feature Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 22:23:40 +02:00
Yury Molodov	5b286a4728	vmui: correct url encoding (#2067) ... * fix: correct encode multi-line queries * fix: change autocomplete for correct arrows work * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:32:30 +02:00
Yury Molodov	d4ccd48faa	vmui: correct calc axes limits (#2058) ... * fix: correct calc axes limits * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:27:26 +02:00
Yury Molodov	e548fc9021	vmui: change layout (#2054) ... * fix: change query reset * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * feat: move time selector to Header * checkout * fix: remove unused vars * update package-lock.json * fix: correct styles * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 12:47:01 +02:00
Aliaksandr Valialkin	cc76174387	app/vmselect/promql: properly keep metric names when optimized path is used for aggregate function calculations ... For example, `sum(rate(...) keep_metric_names) by (__name__)` didn't leave the original metric name because of this issue. This is a follup-up for 1bdc71d917 Udates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-17 15:33:39 +02:00
Aliaksandr Valialkin	5a1e4a140f	app/vmselect/promql: implement keep_metric_names modifier for transform and rollup functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-14 04:15:35 +02:00
Aliaksandr Valialkin	53c9a81f2b	app/vmselect/promql: add stale_samples_over_time() function	2022-01-14 01:49:24 +02:00
Aliaksandr Valialkin	0580a58feb	app/vmselect/promql: add support for @ modifier ... Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier Extend the support with the following features: * Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()` * Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)` returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348	2022-01-13 22:15:19 +02:00
Aliaksandr Valialkin	7d635a2311	app/vmselect/promql: fix limit_offset() test ... The test has been broken in addae7fc6a	2022-01-13 16:58:47 +02:00
Yury Molodov	7ee09286ef	vmui: switching to Preact (#2053) ... * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * fix: remove unused vars * update package-lock.json * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-11 10:34:01 +02:00
Aliaksandr Valialkin	cf5553d18e	app/vmselect/vmui: run make vmui-update after 89ff7b2465	2022-01-03 21:06:34 +02:00
Yury Molodov	85dc41007b	vmui: legend fixes (#1995) ... * feat: add a reset query by clicking the logo * feat: add sequence number for query fields * feat: invert behavior on the graph's legend * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-23 12:15:21 +02:00
Roman Khavronenko	d1c474c5b3	vmselect: update /query_exemplars placeholder (#2000) ... Grafana expects `data` in response to be a slice and logs an err if it is not: ``` err="[]v1.ExemplarQueryResult: decode slice: expect [ or n, but found , error found in #0 byte of ...||..., bigger context ...||..." ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-23 12:08:40 +02:00
Yury Molodov	0952caca24	vmui: add custom start range (#1989) ... * feat: add custom start range * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 20:22:48 +02:00
Yury Molodov	4c9e563c89	vmui: graph fixes (#1982) ... * fix: remove disabling custom step when zooming * feat: add a dynamic calc of the width of the graph * fix: add validate y-axis limits * fix: correct axis limits for value 0 * fix: change logic create time series * fix: change types for tooltip * fix: correct points on the line * fix: change the logic for set graph width * fix: stop checking the period when auto-refresh is enabled * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-20 17:37:45 +02:00
Aliaksandr Valialkin	b393a16a25	app/vmselect/vmui: make vmui-update	2021-12-20 13:50:55 +02:00
Aliaksandr Valialkin	3396f08874	app/vmselect/promql: follow-up after 177e345d8a ... * Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions. * Simplify their implementation * Mention these functions in docs/CHANGELOG.md	2021-12-20 13:15:06 +02:00
匠心零度	cc4725843b	add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972) ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-12-20 13:15:05 +02:00
Aliaksandr Valialkin	c2b0a6109b	app/vmselect/prometheus: typo fix after bf0e3ad2171219b7883b149999c4dc0ac15b9797	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	7c3239966e	app/vmselect/vmui: make vmui-update	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	d9f075622b	app/vmselect: de-duplicate data exported via /api/v1/export/csv by default ... Previously the exported data wasn't de-duplicated. Now it is possible to export the raw data without deduplication by passing reduce_mem_usage=1 query arg to /api/v1/export/csv See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	c170841951	app/vmselect/promql: add timestamp_with_name(m[d]) function ... This function works the same as `timestamp()`, but doesn't remove source time series names. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388	2021-12-15 23:38:09 +02:00
Yury Molodov	603be150c4	vmui: introduce application mode (#1949) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * feat: introduce application mode * build vmui * Revert "build vmui" This reverts commit c0e2415550. * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-15 21:34:04 +02:00
Aliaksandr Valialkin	bc3923111b	lib/storage: return dedup interval in milliseconds from GetDedupInterval() ... This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin	cdfe854c9b	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() ... This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:52:29 +02:00
Yury Molodov	b3389ce26b	vmui: custom step (#1942) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * docs: document that VMUI now supports overriding of `step` query arg, which is passed to `/api/v1/query_range` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-14 14:52:44 +02:00
Yury Molodov	28655e8eb8	vmui: minor fixes (#1936) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document bugfixes Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-13 13:44:32 +02:00
Aliaksandr Valialkin	3d86e6353b	app/vmselect/vmui: make vmui-update	2021-12-13 13:33:35 +02:00
Yury Molodov	6367edae1d	vmui: add a label for the Query field (#1923) ... * feat: add a label for the Query field * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-12 19:10:56 +02:00
Aliaksandr Valialkin	addae7fc6a	app/vmselect/promql: preserve the order of time series passed to limit_offset() function ... Previously time series passed to `limit_offset()` were shuffled according to hash for their labels. This was unexpected behaviour for most users. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951	2021-12-12 18:07:15 +02:00
Yury Molodov	39e29a50f1	vmui: multiple queries (#1916) ... * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * feat: remove save the last query to local storage * fix: handle an error in a table with subqueries * feat: store display type in URL * Revert "feat: store display type in URL" This reverts commit ccc242c69a. * feat: store display type in URL * refactor: move the time setting to a folder * refactor: move the query configurator to a folder * refactor: move the auth settings to a folder * feat: improve styles * feat: add multi query * update package-lock * feat: add display multiple queries * feat: add limits for multiple queries * update dependencies * feat: add history for multiple queries * feat: add line type to legend * feat: change style for switch * feat: change the logic for axes limits for multiple queries * update package-lock.json * update dependencies * feat: add the filter to legend * wip * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-08 16:43:18 +02:00
Aliaksandr Valialkin	54d4a1c959	app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2021-12-06 17:33:49 +02:00
Aliaksandr Valialkin	df810cfc9a	app/vmselect/vmui: make vmui-update	2021-12-06 10:18:03 +02:00
Aliaksandr Valialkin	120fedf1ac	app/vmselect/vmui: make vmui-update	2021-11-30 01:38:59 +02:00
Yury Molodov	500945499f	vmui: optimize render (#1852) ... * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * update package-lock * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-30 00:57:35 +02:00
Aliaksandr Valialkin	648d788a45	app/vmselect/vmui: make vmui-update	2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin	d5ae9e9619	app/vmselect/vmui: make vmui-update	2021-11-29 12:23:12 +02:00
Michael Fuller	8a94dd36d1	vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838) ... Co-authored-by: Michael Fuller <mfuller@digitalocean.com>	2021-11-29 12:14:55 +02:00
Aliaksandr Valialkin	84decc4254	app/vmselect/promql: consistently return zero from deriv(const)	2021-11-17 18:01:58 +02:00
Aliaksandr Valialkin	aa6d11a33f	app/vmselect/promql: add now() function, which returns the current timestamp as a floating-point value in seconds	2021-11-17 16:42:11 +02:00
Aliaksandr Valialkin	1eb756fd11	app/vmselect/promql: arrange function names in the code in alphabetical order ... This should simplify code maintenance in the future	2021-11-14 13:54:43 +02:00
Aliaksandr Valialkin	82cf9f26db	app/vmselect/promql: prevent from incorrect calculations for deriv() over multiple samples with identical timestamps	2021-11-12 13:53:30 +02:00
Aliaksandr Valialkin	4fb19fe34b	all: consistently return application/json content-type without charset=utf-8 ... The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin	31486f2244	app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-09 11:42:43 +02:00
Yury Molodov	89e0f66bfc	vmui: migration MUI Core v4 to v5 (#1795) ... * migration MUI Core v4 to v5 * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-08 15:46:57 +02:00
Aliaksandr Valialkin	3a1ac218ba	app/vmselect/promql: add duration_over_time(m[d], max_interval) function ... This function calculates the actual lifetime of the time series on the given lookbehdind window `d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-08 13:14:57 +02:00
Yury Molodov	654686fd40	vmui: fix graph reset (#1788) ... * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * feat: change graph legend * update dependencies * update codemirror version * fix: correct update period time after zoom/pan * fix: optimize data processing for the graph * fix: eliminate memory leaks related to mouse events * fix: correct display of straight line * Merge branch 'master' into vmui-fix-reset-graph * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-05 20:58:49 +02:00
Aliaksandr Valialkin	be9c7713d7	app/vmselect: make vmui-update	2021-11-04 12:16:55 +02:00
Aliaksandr Valialkin	a102fca4ac	app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778	2021-11-03 16:03:10 +02:00
Aliaksandr Valialkin	43bf0333eb	app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names	2021-11-03 13:20:34 +02:00
Yury Molodov	a16c88095c	vmui: correct migration material-ui (#1758) ... * migration material-ui * fix: rollback popover * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-27 20:43:24 +03:00
Aliaksandr Valialkin	99784b21c1	all: fix build issues and tests for Apple M1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:07:19 +03:00
Yury Molodov	e7591577f0	vmui: update dependencies (#1754) ... * update dependencies * update codemirror version * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-26 19:32:31 +03:00
Aliaksandr Valialkin	d8ce86d685	app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738	2021-10-24 13:32:25 +03:00
Yury Molodov	3ccc37a98b	vmui: query history (#1732) ... * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:29:27 +03:00
Yury Molodov	744f6a98eb	vmui: features (#1711) ... * feat: initial uPlot graph * feat: add zoom/pan for graph * fix: add zoom by ctrl/mac * fix: remove unused code * feat: add toggle cache for fetch * feat: add fix y-axis limits * fix: stop point events while panning * fix: change getting cursor position when scaling * feat: add cursor tooltip to graph * fix: uninstall chart.js * fix: change link for create an issue * fix: set default cache value to true * app/vmalert: follow-up after 0e2486df56 * docs/CHANGELOG.md: document 5416e18007 * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-18 15:19:01 +03:00
Aliaksandr Valialkin	ed994873fd	app/vmselect/promql: randomize the static selection of time series returned from limitk() ... Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series. Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.	2021-10-16 21:16:34 +03:00
Aliaksandr Valialkin	da0adf446d	app/vmselect/promql: typo fix in comment: didsn't -> didn't	2021-10-16 13:00:10 +03:00
Aliaksandr Valialkin	48c3668b3d	app/vmselect/promql: add atan2 binary operator, which is going to be added in Prometheus 2.31 ... See https://github.com/prometheus/prometheus/pull/9248	2021-10-11 21:17:23 +03:00
Aliaksandr Valialkin	b3d28f3872	app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31 ... See https://github.com/prometheus/prometheus/issues/9233	2021-10-11 21:17:21 +03:00
Aliaksandr Valialkin	f1e1d20ac4	app/vmselect/promql: consistently return the same set of time series from limitk() function ... This is the expected behaviour by most users.	2021-10-08 19:55:29 +03:00
Aliaksandr Valialkin	cf87050d83	app/vmui: follow-up after 7bfb44113e ... * Run `vmui-update` * Document the changes in README.md and CHANGELOG.md	2021-10-08 15:11:09 +03:00
Aliaksandr Valialkin	cbf01f7384	app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow ... The list of functions, which can adjust lookbehind window is more limited than the rest of functions, so it is better from maintainability and readability PoV using the allowlist instead of blocklist.	2021-10-07 13:19:22 +03:00
Aliaksandr Valialkin	7e6ec0bbe9	app/vmselect/promql: return back the behaviour for deriv() function when the lookbehind window doesnt contain enough points ... It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d` contains less than 2 samples in the same way as `rate()` does. This is a follow-up after 3e084be06b .	2021-10-07 12:55:58 +03:00
Roman Khavronenko	0a8804d6aa	app/vmselect: make predict_linear and deriv compatible with Prometheus (#1681) ... Previously, `predict_linear` returned slightly different results comparing to Prometheus. The change makes linear regression algorithm compatible with Prometheus. `deriv` was excluded from the list of functions which can adjust the time window for the same reasons.	2021-10-07 12:55:56 +03:00
Aliaksandr Valialkin	ec6eb03d65	app/vmselect/promql: add topk_last and bottomk_last functions	2021-09-30 13:23:27 +03:00
Roman Khavronenko	95ddfda894	app/vmselect: fix binary comparison func (#1667) ... The fix makes the binary comparison func to check for NaNs before executing the actual comparison. This prevents VM to return values for non-existing samples for expressions which contain bool comparisons. Please see added test for example.	2021-09-30 13:04:10 +03:00
Roman Khavronenko	57eb5ef194	app/vmselect: fix testRowsEqual func NaN checks (#1666) ... It appeared, that `testRowsEqual` NaN comparison was incorrect. The fix caused some tests to fail. Please see the change and tests updated. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-30 13:04:08 +03:00
Yury Molodov	a6037a8ce8	vmui: fixed bug with time range (time zone) (#1661) ... * fix: set date in query string in utc format * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 01:01:27 +03:00
Roman Khavronenko	444a8882f4	app/vmselect: disable time-window adjustment for min/max_over_time funcs (#1658) ... Adjustment results into discrepancy between Prometheus and VM on time windows smaller than scrape interval. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:43:54 +03:00
Roman Khavronenko	f46f847531	app/vmselect: always return zero for stddev func if there is only one value (#1659) ... The fix will always return zero if received set of items consists of one element only, which also means no deviation. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:42:19 +03:00
Roman Khavronenko	72468c9d75	app/vmselect: return NaN instead of 0 for empty value sets (#1660) ... The change affects `count/stddev/stdvar_over_time` funcs and makes them to return NaN instead of zero when there is no datapoints in a time window. This is needed for improving compatibility with Prometheus. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:38:24 +03:00
Roman Khavronenko	c8ac8f44b2	app/vmselect: rm quantile_over_time fast-path optimisations (#1662) ... The removed fast path optimisations weren't consistent with `quantile` function behavior and results into discrepancy. Specifically, results didn't match in cases when: * 0 < phi > 1; * values contain only one element. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:35:59 +03:00
Yury Molodov	80df31b2ee	vmui: use Chart.js as default engine for graph (#1634) ... * feat: add Plotly as default engine for graph * fix: remove unused components * feat: use Chart.js as default engine graph * fix: correct styles for loader * feat: add zoom/pan for chart * feat: add height for chart * fix: remove unused code * fix: remove empty units from duration * fix: change debounce for pan to 500ms * fix: add utility for plugins register globally * fix: optimize render graph * feat: add buffer data for zoom * fix: add limits for zoom in/out * fix: change update data while zooming * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-27 22:27:49 +03:00
Aliaksandr Valialkin	4b3951fd86	app/vmselect/promql: add rollup_scrape_interval(m[d]) function ... It calculates the min, max and avg scrape intervals for m over the given lookbehind window d	2021-09-27 19:21:55 +03:00
Aliaksandr Valialkin	eff31c10ec	app/vmselect/promql: follow-up after 526dd93b32 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612	2021-09-27 18:57:03 +03:00
Roman Khavronenko	6061464d80	app/vmselect: quantile func compatiblity with Prometheus (#1646) ... * app/vmselect: `quantile` func compatiblity with Prometheus The `quantile` func was previously calculated by https://github.com/valyala/histogram package. The result of such calculation was always the closest real value to requested quantile. While in Prometheus implementation interpolation is used. Such difference may result into discrepancy in output between Prometheus and VictoriaMetrics. This commit adds a Prometheus-like `quantile` function. It also used by other functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmselect: `quantile` review fixes * quantile functions were split into multiple to provide different API for already sorted data; * float64sPool is used for reducing allocations. Items in pool may have different sizes, but defining a new pool was complicates due to name collisions; Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-27 18:57:02 +03:00
Aliaksandr Valialkin	a746fe8778	app/vmselect/promql: follow-up after 57b3320478	2021-09-24 01:25:04 +03:00
Roman Khavronenko	7ac0559d4f	app/vmselect: make sorting for query result similar to Prometheus (#1647) ... * app/vmselect: make sorting for query result similar to Prometheus Updated sorting allows to get the order of series in result similar or equal to what Prometheus returns. The change is needed for compatibility reasons. * Update app/vmselect/promql/exec_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-24 01:25:03 +03:00
Aliaksandr Valialkin	1115c2f235	app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus ... Previously VictoriaMetrics could return unexpected result of the right-hand side operand had multiple time series with the given set of labels mentioned in `on(labels)`. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643	2021-09-24 00:56:35 +03:00
Aliaksandr Valialkin	3f1104ea74	app/vmselect: fix accessing /graphite/* endpoints	2021-09-21 14:06:18 +03:00
n4mine	a5fcc9e356	fix: typo, dddresses -> addresses (#1630)	2021-09-20 14:27:34 +03:00
Aliaksandr Valialkin	76c650bbda	app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once	2021-09-17 23:35:04 +03:00
Aliaksandr Valialkin	81d24045c2	app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function ... This function calculates multiple quantiles over the given buckets at once Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-09-17 13:39:06 +03:00
Aliaksandr Valialkin	4ed7548ced	app/vmselect/promql: optimize quantiles() calculation ... Calculate quantiles in one go instead of calculating each quantile individually	2021-09-17 12:38:21 +03:00
f41gh7	348f705d95	follow-up after e2f99214f3e518f79c1c96ff918820a69d350ba1	2021-09-16 13:56:28 +03:00
Roman Khavronenko	05bf6cca82	vmselect: add stub for graphite/functions requests ... The request `graphite/functions` is sent by Grafana datasource and results into error logs in vmselect.	2021-09-16 13:51:37 +03:00
Aliaksandr Valialkin	3f8de4f82f	app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL	2021-09-16 13:35:38 +03:00
Aliaksandr Valialkin	a4eaf1a0a1	app/vmselect/prometheus: make more clear log messages for errors during sending data to remote clients	2021-09-16 13:35:37 +03:00
Aliaksandr Valialkin	7ad54041fe	app/{vminsert,vmselect}: automatically add missing port in -storageNode lists passed to vminsert and vmselect ... This should simplify manual setup of the cluster according to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup	2021-09-15 18:04:30 +03:00
Aliaksandr Valialkin	794d941125	app/vmui: follow-up after 777ff75874 ... The commit contains the following changes: - Show vmui when requesting /graph page in order to be compatible with Prometheus datasource in Grafana. - Properly encode query args at vmui url. - Set the number of points on the graph to the number of horizontal pixels divided by 2. Previously it was hardcoded to 30. - Do not save server url to persistent storage at browser, since it should be always obtained from the url. - Run `make vmui-update` for updating vmui embedded into VictoriaMetrics.	2021-09-15 17:41:00 +03:00
Aliaksandr Valialkin	19516aedb5	app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function	2021-09-13 13:48:18 +03:00
Aliaksandr Valialkin	9b07cb2988	app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600	2021-09-09 19:24:10 +03:00
Aliaksandr Valialkin	3788d4f4eb	app/vmselect: show useful endpoints when requested /select/<accountID>/ page	2021-08-29 12:05:14 +03:00
Aliaksandr Valialkin	39bb6bdd79	app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-08-27 18:40:25 +03:00
Aliaksandr Valialkin	335de30083	app/vmselect/promql: make fmt after 0078486ea7	2021-08-23 23:05:34 +03:00
Aliaksandr Valialkin	40b06e84f8	app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus ... See https://github.com/prometheus/prometheus/pull/8457 for details.	2021-08-23 11:46:29 +03:00
Aliaksandr Valialkin	91534057a3	app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d]) ... This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d	2021-08-21 19:56:56 +03:00
Roman Khavronenko	1ccb77904b	vmselect: update vm_request_duration_seconds value when request fails (#1537) ... Before, metric `vm_request_duration_seconds` was update only on successful attempts which could be misleading. For example, timeout errors on netstorage request may be not accounted in the metric and won't be visible on dashboards. Using `defer` statement to update the metric after query arguments validation may improve the situation.	2021-08-19 14:07:00 +03:00
Aliaksandr Valialkin	ee1f3414d1	app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval ... Since most users do not expect the overriding of explicitly set `[d]`.	2021-08-19 10:33:10 +03:00
Aliaksandr Valialkin	5d92fafc40	app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries ... This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin	49886ecbc8	app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541	2021-08-17 13:22:15 +03:00
Aliaksandr Valialkin	38065bec7b	app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability	2021-08-17 11:00:58 +03:00
Aliaksandr Valialkin	5420c3d967	app/vmselect/promql: drop staleness marks before calling rollupConfig.Do ... This allows dropping staleness marks only once and then calculate multiple rollup functions on the result. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-15 13:22:26 +03:00
Aliaksandr Valialkin	6c4c54eaad	Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions" ... This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346. It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc	2021-08-15 13:22:24 +03:00
Aliaksandr Valialkin	af4a306d7b	app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions ... Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values, which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-14 12:45:31 +03:00
Aliaksandr Valialkin	c1f81f08d4	all: add support for Prometheus staleness markers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin	b35ae791f1	app/vmselect: make vmui-update after the commit 4ae14df864a7e327955f44941295a286175423b3	2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin	60cfa5f100	app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 ... See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097	2021-08-03 12:21:53 +03:00
Nikolay	3f3ad13753	adds /rules and /alerts api for grafana (#1504) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-08-02 17:29:49 +03:00
Aliaksandr Valialkin	99004a6a40	app/vmselect/netstorage: unpack time series data in mostly local big chunks ... This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples	2021-07-30 12:26:33 +03:00
Aliaksandr Valialkin	cbb81c2ce9	app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17	2021-07-30 10:28:19 +03:00
Aliaksandr Valialkin	b709fa387a	app/vmselect: follow-up for ed95bc9531 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493	2021-07-29 09:48:47 +03:00
arnoldyahad	ed95bc9531	Add case prometheus/rules for grafana 8 (#1502)	2021-07-29 06:15:35 +03:00
Aliaksandr Valialkin	49bf3abf67	app/vmselect: follow-up for 626073bca8 ... * Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries * Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes * Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md	2021-07-28 18:00:04 +03:00
Aliaksandr Valialkin	5d255846ac	all: add go:build lines for Go1.17 ... See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin	3921d8afae	app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel	2021-07-26 15:50:45 +03:00
Aliaksandr Valialkin	c3e6ce1db9	app/vmselect: make vmui-update after a91d41f12a	2021-07-26 10:32:01 +03:00
Aliaksandr Valialkin	b047feeb8b	app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series ... The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.	2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin	b92702f6d5	app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores	2021-07-16 00:35:36 +03:00
Aliaksandr Valialkin	df117f85bd	app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries ... This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries or queries that take too much cpu time.	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	5830ce2706	app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series ... This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	6c42db87a8	app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems	2021-07-15 15:40:53 +03:00
Aliaksandr Valialkin	9add9d86a6	app/vmselect/promql: duration handling improvements in MetricsQL queries ... - Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600 - Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])	2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin	f5fa177141	Revert "app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui" ... This reverts commit 885a79def6799f288e14df05b35a12569659ab85. Reason for revert: Grafana doesn't allows accessing /select/<accountID>/prometheus/vmui :(	2021-07-12 09:08:43 +03:00
Aliaksandr Valialkin	ddaa12050d	app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui ... The /select/<accountID>/prometheus/vmui is needed for accessing via server-side Prometheus datasource for Grafana. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-10 12:52:25 +03:00
Aliaksandr Valialkin	0b98f6c7ff	app/vmselect: expose vmui at /vmselect/<accountID>/vmui/ instead of /vmselect/<accountID>/prometheus/vmui/ ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-10 12:32:21 +03:00
Aliaksandr Valialkin	98e049ba6d	app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-09 17:13:51 +03:00
Aliaksandr Valialkin	2c5e1cd893	app/vmselect: move web ui from /ui to /select/<accountID>/prometheus/ui ... This way the UI is available for every tenant (aka accountID) and the UI can automatically determine the needed per-tenant datasource path from page referer. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-08 13:14:50 +03:00
Aliaksandr Valialkin	acb7a95c64	app/vmselect: follow-up after aa11ef6d3b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4	2021-07-07 17:45:09 +03:00
tony	aa11ef6d3b	add vmui for vmselect component (#1431) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-07-07 17:04:23 +03:00
Aliaksandr Valialkin	9c19719ad6	app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms	2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin	ceda2b1df4	lib/httpserver: print full requestURI in httpserver.Errorf ... This should simplify debugging.	2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin	44855f0c9b	app/{vmselect,vmstorage}: clarify the description for -dedup.minScrapeInterval command-line flag ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1426	2021-07-02 15:06:41 +03:00
Aliaksandr Valialkin	856aecae05	app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect	2021-06-23 14:18:37 +03:00
Aliaksandr Valialkin	b92d110cad	app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration	2021-06-18 19:07:03 +03:00
Roman Khavronenko	a15c947045	promql: fix increase_pure calculation for cases with stale series (#1381) ... Due to staleness handling, increase_pure were using incorrect previous value during calculation in cases where series disappears for period longer than staleness period and then returns back. The fix suppose to account for a real datapoint value before staleness takes place. The fix should remove unexpected spikes while using `increase_pure` for staled series.	2021-06-15 17:37:51 +03:00
John Belmonte	3e79f3994e	spelling fix: synonym (#1363)	2021-06-11 10:58:48 +03:00
Aliaksandr Valialkin	e8e7f03394	app/vmselect/promql: typo fix in the comment	2021-06-09 18:34:57 +03:00
Aliaksandr Valialkin	a0b001bfec	app/vmselect/netstorage: remove duplicate limiter on concurrent queries ... It duplicates the `-search.maxConcurrentRequests` limiter.	2021-05-24 19:13:04 +03:00
Aliaksandr Valialkin	180829b8c2	app/vmselect/promql: add timezone_offset(tz) function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306	2021-05-20 11:54:06 +03:00
Nikolay	be87be34a4	Adds tsdb match filters (#1282) ... * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin	56b08390f6	app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode ... `vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.	2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin	6dc5d3b357	all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com	2021-04-20 20:20:01 +03:00
Aliaksandr Valialkin	e48fd5776b	app/vmselect/promql: fix tests after d3fa0ccabd	2021-04-08 00:17:40 +03:00
Aliaksandr Valialkin	f90bf265f4	app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-08 00:10:33 +03:00
Aliaksandr Valialkin	9ce3b7e1dd	app/vmselect: return data:null instead of data:[] from /api/v1/query_exemplars, since Grafana throws an error otherwise ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-07 23:34:01 +03:00
Aliaksandr Valialkin	3d5f1f779f	app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-07 14:16:02 +03:00
Aliaksandr Valialkin	7fce4e9fb4	app/vmselect: return dumb response on /api/v1/query_exemplars request ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-05 23:28:21 +03:00
Aliaksandr Valialkin	2585058a5f	Makefile: prepare arm64 and amd64 release archives for cluster version on make release command	2021-04-05 23:01:45 +03:00
Aliaksandr Valialkin	4aa59cae7c	app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label)	2021-04-03 22:06:50 +03:00
Aliaksandr Valialkin	b9469de410	app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value")	2021-04-02 23:56:30 +03:00
Aliaksandr Valialkin	4028d692f5	app: do not process non-GET requests on at / handler	2021-04-02 22:56:38 +03:00
Aliaksandr Valialkin	31d3d02d25	app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users ... Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.	2021-04-02 16:50:02 +03:00
Aliaksandr Valialkin	d8c769e6af	app/vmselect: add -search.maxStatusRequestDuration command-line flag for limiting the duration of requests to /api/v1/status/* and /api/v1/series/count	2021-03-30 21:39:32 +03:00
Aliaksandr Valialkin	0e29d48628	app/vmselect: remove mentioning for -search.storageTimeout from command-line flag descriptions ... This is a follow-up for ecfd6fe78d	2021-03-30 15:07:40 +03:00
Aliaksandr Valialkin	ecfd6fe78d	app/vmselect: remove -search.storageTimeout command-line flag, since it has the same meaning as -search.maxQueryDuration ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 14:54:36 +03:00
Aliaksandr Valialkin	7bafaad46d	app/vmselect: prevent from possible incomplete query results after timed out query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 13:34:57 +03:00
Nikolay	022fe4efd0	makes tenant metrics names consistent with enterprise version (#1152)	2021-03-29 11:59:04 +03:00
Aliaksandr Valialkin	698ccca6ad	app/vmselect/prometheus: explain why partial responses are disallowed during data export ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148	2021-03-25 18:39:07 +02:00
Aliaksandr Valialkin	b473c21915	app/vmselect/promql: do not merge time series during requests to /api/v1/query ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141	2021-03-25 13:56:23 +02:00
Aliaksandr Valialkin	d1e773266f	app/vmselect/graphite: accept and enforce extra_label in all the Graphite APIs	2021-03-23 15:30:15 +02:00
Aliaksandr Valialkin	27bb614016	app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon	2021-03-23 14:57:57 +02:00
Aliaksandr Valialkin	7f9863254d	app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag	2021-03-22 16:44:36 +02:00
Aliaksandr Valialkin	92531a38c4	app/vmselect/promql: increment key prefix for faster reset for rollup result cache	2021-03-22 11:59:39 +02:00
Aliaksandr Valialkin	799461d8bf	app/vmselect/graphite: follow-up after 529d7be26b	2021-03-18 16:33:00 +02:00
Nikolay	8807410a00	changes metricsFind api (#1137) ... it should be able mitigate crash if label value contains *,[ or { symbols	2021-03-18 16:32:59 +02:00
Aliaksandr Valialkin	667d129e1e	app/vmselect/promql: typo fix after 9666834045	2021-03-17 15:11:54 +02:00
Aliaksandr Valialkin	69201806f8	app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function ... This should improve accuracy for the returned buckets	2021-03-17 14:31:29 +02:00
Aliaksandr Valialkin	05e301cfa0	app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores	2021-03-16 21:52:05 +02:00
Aliaksandr Valialkin	f54ece438d	app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time()	2021-03-16 12:23:55 +02:00
Aliaksandr Valialkin	e2717d84c0	all: various fixes in command-line flag descriptions	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	30ab0eec27	app/vmselect/promql: reduce overhead on scrape interval estimation ... It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.	2021-03-15 20:32:35 +02:00
Aliaksandr Valialkin	8844b38745	app/vmselect/promql: fix tests after 2dae0a2c47	2021-03-15 20:20:27 +02:00
Aliaksandr Valialkin	6b9bba7448	app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point	2021-03-15 12:38:59 +02:00
Aliaksandr Valialkin	b6dcb37fca	app/vmselect/prometheus: do not include datapoints with timestamps matching t-d when returning results from /api/v1/query?query=m[d]&time=t as Prometheus does	2021-03-12 12:18:37 +02:00
Aliaksandr Valialkin	942890b1bb	app/vmselect/promql: follow up for 433fff0006	2021-03-09 12:54:23 +02:00
Nikolay	28e450cd7c	duplicate timeseries fix for prometheus_buckets function (#1119) ... * try fix for prometheus_buckets * merge possible end of the bucket collision	2021-03-09 12:54:21 +02:00
Aliaksandr Valialkin	5277507932	app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL	2021-03-04 14:12:52 +02:00
Aliaksandr Valialkin	3b1c2f03c3	app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration ... This should help debugging slow queries.	2021-02-28 19:42:29 +02:00
Nikolay	673b10dd7f	adds enforced tag filters into cache key (#1095)	2021-02-27 00:23:38 +02:00
Aliaksandr Valialkin	e38ca28d99	app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge ... The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets, then the resulting accuracy can suffer.	2021-02-26 22:56:04 +02:00
Aliaksandr Valialkin	0144b164c7	app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release ... See https://twitter.com/roidelapluie/status/1363428376162295811 The `last_over_time(m[d])` function already exists in MetricsQL.	2021-02-24 17:26:27 +02:00
Aliaksandr Valialkin	1b90a091cf	app/vmselect/netstorage: cleanup after 4805b80977	2021-02-24 11:43:09 +02:00
weng zhao	4805b80977	vmselect: fix vmselect cache partial result when timeout (#1085)	2021-02-24 11:39:56 +02:00
Aliaksandr Valialkin	d16effc29e	app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does	2021-02-24 00:41:30 +02:00
Aliaksandr Valialkin	5dbe88a1c6	app/vmselect/promql: add increase_pure() function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962	2021-02-22 19:15:31 +02:00
Aliaksandr Valialkin	bd1d906eee	app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics	2021-02-18 22:08:13 +02:00
Aliaksandr Valialkin	3062ff0fdb	app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration ... The metrics are: - vm_vmselect_http_requests_total{accountID="...",projectID="..."} - the total number of select requests per each tenant - vm_vmselect_http_duration_ms_total{accountID="...",projectID="..."} - the total duration in milliseconds for per-tenant select requests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932	2021-02-16 23:30:29 +02:00
Aliaksandr Valialkin	73c9da16b8	app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations	2021-02-16 16:09:00 +02:00
Aliaksandr Valialkin	a3a09a3c6e	app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key ... This should prevent from a memory allocation and a string copy.	2021-02-16 15:46:31 +02:00
Aliaksandr Valialkin	46e98ed490	vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 ... The new version switches from log-linear histograms to log-based histograms, which provide up to 3.6 times better accuracy.	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	c769f8321d	deployment/docker: embed tzdata into prod Go app instead of installing it into base docker image ... While this increases app size by 700Kb, this allows using -loggerTimezone in a scratch base image See https://github.com/golang/go/issues/38017	2021-02-12 04:56:27 +02:00
Aliaksandr Valialkin	ff7850aec0	deployment/docker: use docker buildx for creating multiarch builds ... See https://github.com/docker/buildx/	2021-02-12 04:35:35 +02:00
Aliaksandr Valialkin	ee4288987b	app/vmselect/prometheus: treat match query arg in the same way as match[] query arg	2021-02-11 15:01:55 +02:00
Aliaksandr Valialkin	54d2f67924	app/vmselect/promql: properly make copies of EvalConfig	2021-02-11 12:41:37 +02:00
Aliaksandr Valialkin	5973fd4067	app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2 ... This should prevent from data races if the underlying functions modify EvalConfig contents.	2021-02-10 23:06:58 +02:00
Aliaksandr Valialkin	985c3e301d	app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel ... This should reduce query execution times.	2021-02-10 23:00:56 +02:00
Aliaksandr Valialkin	8de3a329ff	app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax	2021-02-03 20:36:42 +02:00
Aliaksandr Valialkin	8249f13104	app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards ... Example: foo{bar{x,yz},a[b-c],*de}	2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin	4b930b9ffe	app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax	2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin	7d23f3ff3a	app/vmselect: add ability to set an additional label filters via extra_label query arg	2021-02-01 17:42:35 +02:00
Nikolay	b8bc1c2e0f	Graphite vmalert wip (#112) ... * init implementation for graphite alerts * adds graphite support for vmalert * small fix * changes vmalert graphite api with type * updates tests * small fix * fixes graphite parse * Fixes graphite from time	2021-02-01 15:28:30 +02:00
Aliaksandr Valialkin	5328a102e0	app/vmselect: unconditionally deny partial responses from /api/v1/export* ... It is expected that `/api/v1/export*` returns full data, so there is no sense in partial responses there.	2021-01-27 14:39:53 +02:00
Aliaksandr Valialkin	44c74f1e79	app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag ... This should reduce incorrect usage and assumptions for this flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020	2021-01-22 13:00:35 +02:00
Aliaksandr Valialkin	e55205220b	app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time	2021-01-19 22:57:50 +02:00
Aliaksandr Valialkin	5856611291	app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()	2021-01-18 17:31:27 +02:00
Aliaksandr Valialkin	c5bdab5a4c	app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992	2021-01-13 12:43:47 +02:00
Aliaksandr Valialkin	8cae98aa78	app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990	2021-01-13 10:59:32 +02:00
Aliaksandr Valialkin	df6e399f73	app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d	2021-01-12 16:12:47 +02:00
Aliaksandr Valialkin	47872ada7e	app/vmselect/promql: do not ajdust offset value provided in the query ... Previously it could be modified in order to improve response cache hit ratio. This is unneeded, since cache hit ratio should remain good because the query time range should be already aligned to multiple of `step` values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976	2020-12-27 14:10:15 +02:00
Aliaksandr Valialkin	5bbf200de2	app/vmselect: add per-tenant /api/v1/status/top_queries handler	2020-12-27 12:53:50 +02:00
Aliaksandr Valialkin	0e739efc88	app/vmselect/promql: simplify defer call for querystats.RegisterQuery	2020-12-27 12:07:56 +02:00
Aliaksandr Valialkin	44932098b5	app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K ... This should reduce memory usage in constrained environments	2020-12-25 17:40:32 +02:00
Aliaksandr Valialkin	e6deb39064	app/vmselect: refactor /api/v1/stats/top_queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907	2020-12-25 17:24:25 +02:00
Nikolay	76d092c091	Adds query stats handler (#945) ... * Adds query stat handler, for query and query_range api, victoriametrics tracks query execution time, stats are expored at /api/v1/status/queries endpoint with topN param https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907 * fixed query stats bugs * improves queryStats tracker * improves query stat * small fix * fix tests * added more tests * fixes 386 tests * naming fixes * adds drop for outdated records	2020-12-25 17:24:24 +02:00
Nikolay	0b87f02602	fixes panic (#979) ... * fixes panic https://github.com/VictoriaMetrics/helm-charts/issues/89 * add fast-path * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-24 09:01:40 +02:00
Aliaksandr Valialkin	262cf81757	app/vmselect: properly parse negative combined offsets such as -1h2m3s ... Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.	2020-12-19 01:25:03 +02:00
Aliaksandr Valialkin	8d1031c29a	app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100	2020-12-15 14:10:50 +02:00
Aliaksandr Valialkin	a2eb451de4	app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser	2020-12-14 14:13:01 +02:00
Aliaksandr Valialkin	d6f9bf2d19	app/vmselect/graphite: properly handle wildcards and charsets inside curly braces ... For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952	2020-12-11 17:26:32 +02:00
Aliaksandr Valialkin	1a237c6903	all: properly handle CPU limits set on the host system/container ... This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin	bd8eef2528	app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect	2020-12-08 15:17:05 +02:00
Aliaksandr Valialkin	9660774fd1	app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942	2020-12-07 01:10:02 +02:00
Aliaksandr Valialkin	d242c2f2bd	app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N	2020-12-05 12:31:01 +02:00
Aliaksandr Valialkin	bdac2171f1	all: do not print usage info for all the flags when incorrect command-line flag is passed ... This should improve usability for VictoriaMetrics apps that have big number of command-line flags, i.e. all the apps.	2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin	8cf76d8747	app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936	2020-12-03 21:46:18 +02:00
Aliaksandr Valialkin	11bbb3552d	app/vmselect/promql: make fmt	2020-12-02 21:34:15 +02:00
Aliaksandr Valialkin	9e98a8f3d3	app/vmselect/promql: return nan from minute(m) when m equals to nan ... This aligns VictoriaMetrics behaviour with Prometheus behaviour. The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 20:16:40 +02:00
Aliaksandr Valialkin	def513355e	app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. ... This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.	2020-12-02 13:12:33 +02:00
Aliaksandr Valialkin	490c70a958	app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true ... This aligns MetricsQL behavior to Prometheus' one. The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 12:09:40 +02:00
Aliaksandr Valialkin	4ef7158e89	app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does	2020-12-02 00:28:56 +02:00
Aliaksandr Valialkin	adf45b730c	app/vmselect/searchutils: return elapsed time in Deadline.String() output ... This should improve debuggability for error messages containing Deadline.String() output	2020-12-01 00:14:36 +02:00
Aliaksandr Valialkin	e1297c0b78	app/vmselect: add /tags/delSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb	2020-11-23 15:32:14 +02:00
Aliaksandr Valialkin	3d2ce31cad	app/vmselect/netstorage: code readability improvement: rename *RequestErrors to *Errors	2020-11-23 15:00:15 +02:00
Aliaksandr Valialkin	433ae806ac	app/vmselect: implement /tags/tagSeries and /tags/tagMultiSeries` in order to be consistent with single-node VictoriaMetrics	2020-11-23 14:57:08 +02:00
Aliaksandr Valialkin	7987129baa	app/vmselect/netstorage: move common code for requests execution on all the storage nodes to startStorageNodesRequest func	2020-11-23 10:51:48 +02:00
Aliaksandr Valialkin	25a57ced6c	app/vmselect/netstorage: prevent from data races in ProcessSearchQuery and in Export funcs when -replicationFactor > 1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-11-23 10:25:51 +02:00
Aliaksandr Valialkin	f4fd917e4f	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin	1dcb438c3b	app/vmselect/netstorage: typo fix after 990eb29a9b	2020-11-23 01:09:43 +02:00
Aliaksandr Valialkin	85eecf5801	app/vmselect/netstorage: add -replicationFactor command-line flag for reducing query duration when a part of vmstorage nodes are temporarily slow and/or temporarily unavailable ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-11-23 00:39:53 +02:00
Aliaksandr Valialkin	990eb29a9b	app/vmselect/netstorage: move common code for collecting query results from vmstorage nodes to collectResults function	2020-11-23 00:16:02 +02:00
Aliaksandr Valialkin	e72ccc9239	app/vmselect: add remoteAddr to slow query log in order to improve debuggability ... This will simplify identifying the client that sends slow queries to VictoriaMetrics.	2020-11-18 20:40:02 +02:00
Aliaksandr Valialkin	ea4afb201b	app/vmselect/netstorage: typo fix in a comment inside SearchMetricNames func	2020-11-18 01:35:37 +02:00
Aliaksandr Valialkin	7d76fdedcc	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place ... This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin	911c6d3bcd	app/vmselect: add missing graphite prefix to /tags/autoComplete/{tags,values}	2020-11-16 18:04:24 +02:00
Aliaksandr Valialkin	f7f866d83b	app/vmselect/netstorage: typo fix	2020-11-16 15:54:45 +02:00
Aliaksandr Valialkin	59fb75717e	app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name	2020-11-16 15:50:53 +02:00
Aliaksandr Valialkin	eb763bcb9d	app/vmselect/graphite: add /tags/autoComplete/values handler from Graphite Tags API	2020-11-16 15:29:29 +02:00
Aliaksandr Valialkin	f2f16d8e79	app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support	2020-11-16 14:58:10 +02:00
Aliaksandr Valialkin	2f4421b86c	app/vmselect/prometheus: return __name__ label if match[] query to /api/v1/labels matches at least a single time series	2020-11-16 13:54:50 +02:00
Aliaksandr Valialkin	852aed62f7	app/vmselect/prometheus: improve performance for /api/v1/labels and /api/v1/label/<labelName>/values on time ranges exceeding one day when match[] query arg is set	2020-11-16 13:46:51 +02:00
Aliaksandr Valialkin	e969346e3e	app/vmselect/prometheus: fix deadlock in /api/v1/series on a time range exceeding one day	2020-11-16 13:30:57 +02:00
Aliaksandr Valialkin	eea1be0d5c	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:52:23 +02:00
Aliaksandr Valialkin	97100b1d42	app/vmselect/graphite: apply filter then limit	2020-11-16 12:52:18 +02:00
Aliaksandr Valialkin	5889273920	app/vmselect/graphite: add /tags/<tag_name> handler for Graphite Tags API	2020-11-16 03:41:41 +02:00
Aliaksandr Valialkin	99cb1a70cf	app/vmselect/graphite: add /tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 02:57:20 +02:00
Aliaksandr Valialkin	2ac5f00d98	app/vmselect: propagate errors from vmstorage to response to the client if -search.denyPartialResponse command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891 This commit also adds `"isPartial":{true|false}` field to `/api/v1/*` responses. `"isPartial":true` is set when the response is based on a partial data because some of vmstorage nodes weren't available during query processing.	2020-11-14 13:20:10 +02:00
Aliaksandr Valialkin	8f42e59e05	app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points ... Do not take into account spare data point value if the next point will is located too far from the current point. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 15:23:37 +02:00
Aliaksandr Valialkin	da6d82a8dd	app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() ... This should remove unexpected spikes at the end of gaps. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 14:59:32 +02:00
Aliaksandr Valialkin	7ceaf4ba8f	all: consistently return text-based HTTP responses with charset=utf-8 ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:30:21 +02:00
Aliaksandr Valialkin	4f2c5877db	app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters	2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin	a78bf34ff3	app/vmselect: do not return isPartialResponse=true when all the storageNodes return errors	2020-11-10 18:48:57 +02:00
Aliaksandr Valialkin	8f3339fa81	app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin	6385432611	app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum	2020-11-10 01:18:59 +02:00
Aliaksandr Valialkin	efebc3b6fb	app/vmselect/promql: code cleanup after 43823addea	2020-11-06 01:31:33 +02:00
n4mine	3127aa92b5	app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883)	2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin	72011bcc45	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin	c5e6c5f5a6	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin	66de02fbb4	app/vmselect/promql: allow dropping trailing sample only for default_rollup function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850	2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin	6b623eba02	app/vmselect/promql: go fmt	2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin	7c0b658865	app/vmselect/promql: do not drop trailing datapoints for instant queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-11-02 00:12:53 +02:00
Aliaksandr Valialkin	c4464594b7	app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K	2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin	ee2902ddaf	app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series ... Now trailing data points are additionally dropped for time series with a single raw sample Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-17 10:44:26 +03:00
Aliaksandr Valialkin	b9a4601c97	app/vmselect/promql: return a single time series at max from absent() function like Prometheus does	2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin	217c192c88	app/vmselect/promql: improve time series staleness detection ... This should prevent from double counting for time series at the time when it changes label. The most common case is in K8S, which changes pod uid label with each new deployment. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin	f877e703c8	app/vmselect/promql: fix mode_over_time calculations ... Previously `mode_over_time` could return garbage due to improper shuffling of input data points.	2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin	d884ab13dc	app/vmselect/prometheus: fix golangci-lint warning	2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin	0867dea5fc	app/vmselect: add ability to export data in CSV format via /api/v1/export/csv	2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin	2d03d0e2dd	app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning ... Functions are: * keep_last_value * keep_next_value * interpolate * running_min * running_max * running_avg * range_min * range_max * range_avg * range_first * range_last * range_quantile * smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	3881c84afe	Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" ... This reverts commit ac45082216. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric names: * ceil * floor * round Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	79d70480b7	Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" ... This reverts commit bb61a4769b. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * clamp_min * clamp_max Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	8c37b63ea9	Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" ... This reverts commit e5202a4eae. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * max_over_time * min_over_time * avg_over_time * quantile_over_time * geomean_over_time * mode_over_time * holt_winters * predict_linear Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	f6ee6efc34	app/vmselect/promql: add missing label filters to binary operands before query execution ... This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization See also https://github.com/cortexproject/cortex/issues/3253	2020-10-07 21:17:11 +03:00
Aliaksandr Valialkin	abfd3a8fab	app/{vminsert,vmselect,vmstorage}: add a link to https://victoriametrics.github.io/Cluster-VictoriaMetrics.html from main page of every cluster component	2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin	3e0c473cc9	app/vmselect/promql: fill gaps on graphs for range_* and running_* functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806	2020-10-02 14:02:47 +03:00
Aliaksandr Valialkin	61f4801b93	app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output	2020-09-29 18:03:45 +03:00
Aliaksandr Valialkin	0d082cdf53	app/vmselect/netstorage: remove unused code	2020-09-29 11:55:08 +03:00
Aliaksandr Valialkin	c47972d843	app/vmselect/prometheus: check for errors returned from bufferedwriter.Write ... This makes `make errcheck` happy	2020-09-29 11:36:54 +03:00
Aliaksandr Valialkin	e66f7edfc9	app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.* ... In this case only node must be returned with stripped dot in the end of id as carbonapi does	2020-09-29 11:29:38 +03:00
Aliaksandr Valialkin	6d8c23fdbd	app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native ... This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics	2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin	db14f22fc0	app/vmselect: stop /api/v1/export/* execution if client disconnects	2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin	aadbd014ff	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin	973df09686	app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls	2020-09-24 20:44:15 +03:00
Aliaksandr Valialkin	b8bce348c5	app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784	2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin	8627365b48	app/vmselect/prometheus: code cleanup after 3ba507000c	2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin	1fce79518a	app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin	0468cdf33e	app/vmselect/netstorage: properly pre-allocate space for sbs	2020-09-22 23:51:01 +03:00
Aliaksandr Valialkin	69eb9783e6	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:21:42 +03:00
Aliaksandr Valialkin	e564725641	app/vmselect/searchutils: fixed tests after 2eb72e09ab	2020-09-21 21:31:28 +03:00
Aliaksandr Valialkin	07c6226334	app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:46 +03:00
Aliaksandr Valialkin	a1bebb660c	app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API	2020-09-18 11:53:28 +03:00
Aliaksandr Valialkin	9793008734	app/vmselect: add -search.storageTimeout command-line flag for limiting the maximum duration of query execution per each -storageNode ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:33:47 +03:00
Aliaksandr Valialkin	a9205fe308	app/vmselect: prevent from closing connection to vmstorage on query timeout by setting +2 secs deadline on connection comparing to query deadline ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:14:00 +03:00
Aliaksandr Valialkin	1587f83fa0	app/vmselect/netstorage: typo fix after 03dfccfbed	2020-09-16 00:10:33 +03:00
Aliaksandr Valialkin	03dfccfbed	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:09 +03:00
Aliaksandr Valialkin	27cd5555e6	app/vmselect/netstorage: mention that RunParallel or Cancel must be called on the returned results from ProcessSearchQuery	2020-09-15 20:39:43 +03:00
Aliaksandr Valialkin	1f0c0b0f6b	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:25:34 +03:00
Aliaksandr Valialkin	b776a93608	app/vmselect/promql: support composite durations like Prometheus 2.21 does ... The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin	d3ad0d365e	app/vmselect: move Deadline from netstorage to searchutils ... This removes dependency on netstorage from searchutils.	2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin	579c20756a	app/vmselect: substitute inf values at smooth_exponential with the previous values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin	d67e6d3d2e	app/vmselect: skip infinite values when calculating smooth_exponential ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:57:53 +03:00
Aliaksandr Valialkin	06427a184f	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:59:52 +03:00
Aliaksandr Valialkin	f307e6f432	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin	f3a79abfb4	app/vmselect/promql: go fmt	2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin	4f06eed1c1	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin	0d0b606455	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin	db91045348	app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3	2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin	804304c365	app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true	2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin	478d8f8393	app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions ... These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin	3490160fd0	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	285665e93b	app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)	2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin	6721e47ae9	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin	62b6e54622	app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin	c9f5c5623f	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores ... This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin	95a8c492ef	app/vmselect/promql: properly handle -n^m like Prometheus does ... `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin	7f93d61a56	app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin	01000505a0	app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin	75bff1a567	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus ... Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin	8835004a4c	app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does ... This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin	14ddb8a34e	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block ... This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin	46c98cd97a	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin	a04f4a3d9a	app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration	2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin	bdb881c43b	app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...)	2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin	94471a1273	app: remove duplicate *-pure makefile rules	2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin	a2aa3a60eb	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin	106e302d7a	all: add mssing APP_NAME to vm*-GOARCH builds	2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin	0c00fe70cf	app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin	338ee47d60	app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window ... Just divide the data point value by the window duration in this case.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	717c554fb0	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now ... See e00cfc854d for the approach used now.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	d9037b3970	app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	0f6f0d30d3	app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query	2020-07-28 15:14:40 +03:00
Aliaksandr Valialkin	9dccedc599	app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp ... This aligns `group()` behaviour to Prometheus	2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin	b191e425b3	app/vmselect/promql: improve further the accuracy of buckets_limit() function ... The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin	43871e79c6	app/vmselect/promql: avoid dropping inf bucket in buckets_limit ... The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin	978c1e930e	app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets	2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin	51cbf27077	app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function ... Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin	cf69b1ea6f	app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k ... This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin	45334f61de	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin	3526e8768a	app/vmselect/promql: typo fix after 3e557c9861	2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin	8d1721d128	app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration ... Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin	88e8bed0c9	app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time ... This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin	fb3d1380ac	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin	dbf3038637	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin	16a4b1b20c	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks	2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin	55ed07add1	app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g	2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin	7aa5b48508	app/vmselect: reduce memory usage when querying big number of time series with long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin	20d0c41ac5	app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin	bd4299fafe	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries ... This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile	2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin	5bb4fe1ba4	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin	0755cb3b50	app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value ... This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin	71eba8dcf5	app/vmselect: log the total available memory for concurrent requests on not enough memory errors ... This should simplify root cause analysis	2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin	141e84b5a4	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range ... This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin	4d2011a87d	app/vmselect/promql: add mode() aggregate function	2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin	31ef39e8da	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin	427fa43ce2	app/vmselect/promql: add mode_over_time(m[d]) function ... See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin	eb402a17bd	app/vmselect/promql: optimize group(rollup(m)) calculations	2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin	ea8dc85ba8	app/vmselect/promql: check that any() doesn't touch metric name	2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin	fc8fe38a82	app/vmselect/promql: add group() aggregate function to MetricsQL ... This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin	c64914a7e4	app/vmselect/promql: keep all labels for time series from any() call	2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin	14dc426b45	app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage	2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin	0e7b2008b2	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time ... Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin	3898cc0285	app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9	2020-07-13 21:41:17 +03:00
faceair	bf39e67ade	fix empty response template (#617)	2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin	d449d0a0e1	app/vmselect/promql: add missing tests for ifnot binary operation	2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin	7e706eea13	app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation	2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin	6c1a47b5e0	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin	fb86071552	app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528	2020-07-08 19:09:31 +03:00
DexterZhang	9930ce1fa9	Feat/query list vmselect (#575) ... * feat(vmselect): add support for listing current running queries and canceling specific query * fix(vmselect): change current queries' pid from int64 counter to uuid * feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth * fix(vmselect): add more info to current queries * review: delete some unnecessary code and use function instead of init * review: returen *queriesMap in newQueriesMap * review: delete unused var in struct queriesMap, add comments to exported functions * review: add return if error occurs * feat(vmselect): truncate query string in current running query list API since the size of query string might be large; use query string's pointer in struct `query` for the same reason; add query info API to get full access of query's info;	2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin	82871fb7a5	app/vmselect/prometheus: small fixes on top of 8bb762124a	2020-07-05 18:17:53 +03:00
faceair	17f175ff5a	fix adjust last points avoid influence earlier value (#606)	2020-07-05 18:17:52 +03:00
Aliaksandr Valialkin	f10e8809c0	app/vmselect: add interpolate function for filling gaps with linearly interpolated values ... See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin	4cb3e7595c	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention	2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin	0c4e8aeb2b	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin	d962568e93	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin	12b87b2088	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series ... This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:37:35 +03:00
nicbaz	46c5c0772c	vmselect: fix label_replace when mismatch (#579) ... As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the **MetricsQL** `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:54:29 +03:00
Aliaksandr Valialkin	0fdbe5de25	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series ... Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin	70bf8218bb	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin	6939e36fdd	app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin	cc2225cc49	app/vmselect: fix the error after 936f35920a	2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin	936f35920a	app/vmselect/prometheus: allow returning partial response from /api/v1/export if -search.denyPartialResponse=false ... This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.	2020-06-12 21:11:48 +03:00
Aliaksandr Valialkin	3d0a0b3785	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:14:04 +03:00
DexterZhang	fa103875a0	feat(vmselect): add tmp block dir size metrics vm_tmp_blocks_files_size_total (#527) ... * feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total` * refactor(vmselect): use free space instead of used space in tmp block file metrics * fix: add `bytes` suffix to tmp dir free space metric	2020-06-04 13:05:50 +03:00
Aliaksandr Valialkin	d83c68ca03	app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions ... These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin	8ff28f5b91	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin	7d46dd452a	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin	37068064dd	app/vmselect/promql: fix outilersk calculations	2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin	fc81ea38d4	app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin	2784015a4d	all: print --help output to stdout instead of stderr ... This is easier to grep and pipe	2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin	3845420a8f	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin	f3d9a5b0ec	app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint	2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin	18a0caee43	app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one	2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin	81b8811cf4	app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag ... Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin	408ade27a9	app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y	2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin	21c2982ac8	app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation	2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin	f341c6fcc4	Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors" ... This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c. Reason for revert: this commit has been removed from single-node version	2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin	9ed4951ec8	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin	cd1145e5f4	app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors	2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin	e933cbac16	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin	23a310cc68	app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks ... This should minimize the number of disk seeks when reading data from temporary file.	2020-04-26 16:46:17 +03:00
Aliaksandr Valialkin	32b3f959fc	app/vmselect: fix description for -search.resetCacheAuthKey	2020-04-24 19:44:35 +03:00
Aliaksandr Valialkin	f9526809e5	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin	b59f1f1504	app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin	603d4c9217	app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( ... Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin	b9b5641c2f	app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query	2020-04-17 12:31:18 +03:00
Aliaksandr Valialkin	d4bc60d63c	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:45 +03:00
Aliaksandr Valialkin	a873b553cf	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin	a012f6fe70	app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions	2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin	a53e332a93	app/vmstorage: add missing shutdown for http server on graceful shutdown ... This could result in the following panic during graceful shutdown when `/metrics` page is requested: http: panic serving 10.101.66.5:57366: runtime error: invalid memory address or nil pointer dereference goroutine 2050 [running]: net/http.(*conn).serve.func1(0xc00ef22000) net/http/server.go:1772 +0x139 panic(0xa0fc00, 0xe91d80) runtime/panic.go:973 +0x3e3 github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache.(*Cache).UpdateStats(0x0, 0xc0000516c8) github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache/cache.go:224 +0x37 github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexDB).UpdateMetrics(0xc00b931d00, 0xc02c41acf8) github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:258 +0x9f github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*Storage).UpdateMetrics(0xc0000bc7e0, 0xc02c41ac00) github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:413 +0x4c5 main.registerStorageMetrics.func1(0x0) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:186 +0xd9 main.registerStorageMetrics.func3(0xc00008c380) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:196 +0x26 main.registerStorageMetrics.func7(0xc) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:211 +0x26 github.com/VictoriaMetrics/metrics.(*Gauge).marshalTo(0xc000010148, 0xaa407d, 0x20, 0xb50d60, 0xc005319890) github.com/VictoriaMetrics/metrics@v1.11.2/gauge.go:38 +0x3f github.com/VictoriaMetrics/metrics.(*Set).WritePrometheus(0xc000084300, 0x7fd56809c940, 0xc005319860) github.com/VictoriaMetrics/metrics@v1.11.2/set.go:51 +0x1e1 github.com/VictoriaMetrics/metrics.WritePrometheus(0x7fd56809c940, 0xc005319860, 0xa16f01) github.com/VictoriaMetrics/metrics@v1.11.2/metrics.go:42 +0x41 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.writePrometheusMetrics(0x7fd56809c940, 0xc005319860) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/metrics.go:16 +0x44 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb5a120, 0xc005319860, 0xc005018f00, 0xc00002cc90) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:154 +0x58d github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb5a120, 0xc005319860, 0xc005018f00) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:119 +0x8e net/http.HandlerFunc.ServeHTTP(0xc00002d110, 0xb5a660, 0xc0044141c0, 0xc005018f00) net/http/server.go:2012 +0x44 net/http.serverHandler.ServeHTTP(0xc004414000, 0xb5a660, 0xc0044141c0, 0xc005018f00) net/http/server.go:2807 +0xa3 net/http.(*conn).serve(0xc00ef22000, 0xb5bf60, 0xc010532080) net/http/server.go:1895 +0x86c created by net/http.(*Server).Serve net/http/server.go:2933 +0x35c	2020-04-02 21:09:55 +03:00
Aliaksandr Valialkin	c6cbc0bd19	app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries	2020-03-29 21:56:52 +03:00
Aliaksandr Valialkin	cb8696699a	app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()	2020-03-29 21:50:35 +03:00
Dmitry Naumov	b84071fc25	Rootless docker images by default (#358) ... * Rootless docker images by default * Migrate to rootless base image Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-03-27 21:18:32 +02:00
Aliaksandr Valialkin	58cb7fc476	app/vmselect: adjust label_map() handling for corner cases ... The following corner cases now supported: * label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label` * label_map(q, "label", "foo", "") - removes `label="foo"` from series All the unmatched labels are kept unchanged.	2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin	0e7a71a245	app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin	50555d89d3	app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model	2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin	f6410ff2bf	app/vmselect: add optional max_rows_per_line query arg to /api/v1/export ... This arg allows limiting the number of data points that may be exported on a single line.	2020-03-10 21:47:43 +02:00
Aliaksandr Valialkin	0d893eff36	Makefile: add build and test rules with enabled race detector. These rules have -race suffix ... Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin	d23df53ba2	app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter ... This should reduce query time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 23:36:38 +02:00
Aliaksandr Valialkin	8fa1cd24d8	app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 12:17:30 +02:00
Aliaksandr Valialkin	cf9aee4ec3	all: properly split vm_deduplicated_samples_total among cluster components ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin	6ca1e58d98	app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick	2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin	b58e3fc8a9	app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-23 18:06:26 +02:00
Yaroslav	c69d4b01f0	fix rollupOpen(), rollupHigh(), rollupLow() functions (#328)	2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin	f22aefdb16	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin	d5c2a0ce64	app/vmselect: add -search.cacheTimestampOffset command-line flag ... This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin	c70822db50	app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 14:02:12 +02:00
Aliaksandr Valialkin	846d7fa7e9	app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions ... This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin	6e0013ca39	app/vmselect/prometheus: typo fix in -latencyOffset description	2020-02-12 14:00:38 +02:00
Edouard Hur	e8f92a4ee8	Cluster - prometheus metrics fix (#314) ... * add missing '/{}' in prom query range requests * fix missing leading '/' on prom lavelValuesErrors path	2020-02-10 22:15:21 +02:00
Aliaksandr Valialkin	1010a57882	all: allow setting flags via environment vars ... Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin	ea66212c93	lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version	2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin	e6d9ea3094	app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8	2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin	4a1de7fee9	app/vmselect/promql: properly adjust time range for data to select ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin	8e77b54846	app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent	2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin	ce38b176bc	app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin	4f7116d1ee	app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin	ccd3aa4f15	app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded ... This will prevent from excess CPU usage for timed out queries.	2020-02-04 16:20:48 +02:00
Aliaksandr Valialkin	e6bf88a4d4	app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana ... See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.	2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin	7cde594696	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin	45bc6c62f2	app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus	2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin	e3adc095bd	all: add -dedup.minScrapeInterval command-line flag for data de-duplication ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin	cb5c39ee70	lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor ... This should improve performance when reading many small blocks.	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	170c1c3a4e	app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value	2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin	b28c9a3944	app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value ... Examples for such counters: OS-level counters for network or cpu stats.	2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin	3e304890a6	app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296	2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin	ea53a21b02	all: consistently log durations in seconds with millisecond precision ... This should improve logs readability	2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin	e1a264173a	app/vmselect: mention the original query and time range in error messages ... This should simplify debugging invalid or heavy queries.	2020-01-22 17:34:35 +02:00
Aliaksandr Valialkin	e127173984	app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors	2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin	f3b9f8b823	app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data	2020-01-22 15:53:41 +02:00
Aliaksandr Valialkin	40e564eb9c	app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d	2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin	9eaa2ab871	app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin	54db08a60f	app/vmselect/netstorage: make fmt	2020-01-17 17:46:20 +02:00
Aliaksandr Valialkin	d21cc2d16a	app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file ... This should reduce memory usage on systems with more than 8GB RAM.	2020-01-17 16:28:28 +02:00
Aliaksandr Valialkin	b05f6cf11c	app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs ... A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value. This will give more chances for executing insert path.	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	a9f683423c	app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	4b16b7fd11	all: mention command-line flags used for limiting the incoming request size in error messages ... This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin	ce0b602405	app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args	2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin	bcd3f0c5bd	app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions ... These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin	fc01b11ddc	app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval	2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin	16fb128bbc	app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions ... This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.	2020-01-11 00:26:11 +02:00
Aliaksandr Valialkin	adc36d00b7	app/vmselect/promql: properly handle aggr(aggr_over_time(...))	2020-01-10 21:57:11 +02:00
Aliaksandr Valialkin	87a106702b	app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function ... This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin	c314d9a219	app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions ... These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin	7c6df1e51d	app/vmselect/promql: skip rate calculation for the first point on time series	2020-01-08 14:42:44 +02:00
Aliaksandr Valialkin	76707b2ab9	app/vmselect/promql: fix calculations for histogram_share	2020-01-04 14:44:15 +02:00
Aliaksandr Valialkin	accad01b3e	app/vmselect/promql: add missing MetricName into netstorage.Result in tests	2020-01-04 12:53:14 +02:00
Aliaksandr Valialkin	6f29d37cb5	app/vmselect/promql: add histogram_share(le, buckets) function	2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin	2290503140	app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 ... See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin	67f94bbe12	app/vmselect/promql: add histogram_over_time(m[d]) rollup function	2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin	9a1f6848ca	app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time ... Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.	2020-01-03 20:44:54 +02:00
Aliaksandr Valialkin	3d0c7b095a	app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series ... This should provide better estimation for `rate` in the beginning of time series.	2020-01-03 19:03:32 +02:00
Aliaksandr Valialkin	6ea7f23446	app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series	2020-01-03 01:02:30 +02:00
Aliaksandr Valialkin	e0abf45d45	app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations	2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin	eb1a66c577	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin	453d71d082	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:09:09 +02:00
Mike Poindexter	009d1559db	Split Extended PromQL parsing to a separate library	2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin	ff18101d30	app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range ... This should prevent from the following panic at app/vmselect/promql/binary_op.go:255: BUG: len(leftVaues) must match len(rightValues) and len(dstValues)	2019-12-24 22:45:49 +02:00
Aliaksandr Valialkin	e24ee43109	app/vmselect/promql: adjust calculations for rate and increase for the first value ... These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.	2019-12-24 19:41:03 +02:00
Aliaksandr Valialkin	9a2554691c	app/vmselect/promql: properly calculate rate on the first data point ... It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape, i.e. we can assume its value was 0 at this time.	2019-12-24 15:55:15 +02:00
Aliaksandr Valialkin	97de50dd4c	app/vmselect/netstorage: improve error message when reading data size in readBytes	2019-12-24 14:40:14 +02:00
Aliaksandr Valialkin	6358cf3d47	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:26 +02:00