1041 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	464325a24b	app/vmselect/vmui: make vmui-update after da04e9d1de	2022-04-29 12:54:10 +03:00
Dima Lazerka	ed8e88af11	Export "null" in jsonl instead of NaN (#2518) ... * Export "null" in jsonl instead of NaN The NaN appeared because of staleness markers that were added for compatibility. I think it's better to use json `null`, implemented here. Also maybe it also makes sense to add a flag like `?skip-staleness-markers=true` to `/export`, to skip nulls at all? * Update app/vmselect/prometheus/export.qtpl * app/vmselect/prometheus/export.qtpl.go: `make quicktemplate-gen` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-29 12:49:47 +03:00
Nikolay	c7aad8d441	app/vmselect: adds API /api/v1/status/buildinfo (#2515) ... * app/vmselect: adds API /api/v1/status/buildinfo it should fix an compability error with grafana 8.5 prometheus datasource https://github.com/grafana/grafana/pull/46771 * Update main.go	2022-04-29 11:36:28 +03:00
Aliaksandr Valialkin	cce1b6d7f9	app/vmselect/promql: add tlast_change_over_time(m[d]) function, which returns the timestamp for the last change of m on the given lookbehind window d	2022-04-27 10:59:03 +03:00
Yury Molodov	c7693e8bc1	vmui: expression alias (#2495) ... * feat: add alias for queries * docs: update docs for predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-26 15:59:37 +03:00
Yury Molodov	9b4bff67e0	vmui: add support relative time (#2504) ... * feat: add support relative time * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-26 15:46:06 +03:00
Aliaksandr Valialkin	25fe83577d	app/vmselect/promql: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries ... Previously `vector` time series could be unexpectedly returned from such queries	2022-04-21 15:34:36 +03:00
Aliaksandr Valialkin	d1a9fac894	app/vmselect/promql: fix comparison to nan ... The comparison to nan has been broken in d335cc886c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2022-04-21 14:55:37 +03:00
Aliaksandr Valialkin	de892239a9	app/vmselect/promql: add drop_common_labels() function	2022-04-21 14:20:20 +03:00
Aliaksandr Valialkin	7a44ba1234	app/vmselect/promql: fix q default b where b may have empty time series	2022-04-21 11:42:42 +03:00
Aliaksandr Valialkin	d335cc886c	app/vmselect/promql: fix duplicate time series error on joins against time series filtered by values ... This should prevent from `duplicate time series` errors when executing the following query: kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1) where `kube_pod_status_phase{phase=~"Pending|Running"}==1` filters out diplicate time series	2022-04-20 22:18:44 +03:00
Aliaksandr Valialkin	ed97908ca9	app/vmselect/promql: rename removeNaNs() to more clear removeEmptySeries()	2022-04-20 19:53:46 +03:00
Yury Molodov	514e3660e2	fix: prevent graph hiding without data (#2456) ... * fix: prevent graph hiding without data * fix: add yaxis labels default * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 17:15:31 +03:00
Aliaksandr Valialkin	ef66b048c9	app/vmui: further improvements for number display on graphs ... This is a follow-up for c4d2cd8336 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2409	2022-04-12 16:01:27 +03:00
Aliaksandr Valialkin	7f83dc06c4	app/vmselect: make vmui-update	2022-04-12 14:35:19 +03:00
Aliaksandr Valialkin	61c7f6beae	app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2406	2022-04-11 12:34:43 +03:00
Aliaksandr Valialkin	19ecc4b2c3	app/vmselect: make vmui-update	2022-04-01 12:55:21 +03:00
Yury Molodov	f166f80f15	vmui: grid support for predefined panels (#2386) ... * update packages * feat: add setting width for predefined panels * docs: update doc by predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-01 12:48:17 +03:00
Yury Molodov	c8d29ed78e	vmui: predefined panels (#2243) ... * feat: add basic components for predefined dashboards * fix: change display alert * feat: add autosize and unit for axes * feat: add component for CircularProgress * feat: change layout for predefined dashboards * feat: add override step for predefined panels * feat: add override step for predefined panels * feat: change yaxis limits for predefined panels * fix: rename flag for hide legend * feat: add formatted panel description * feat: add README.md for dashboard setup * feat: validate dashboard settings * feat: add unit for y-ticks * fix: correct display error for dashboards * fix: disable auto refresh after route change * update package-lock.json * fix: add basename for BrowserRouter * fix: add dynamic basename for routing * update packages * feat: add a pre-defined dashboard "per-job resource usage" * feat: display unit in the hover-tooltip * fix: change routing and home layout * fix: change axis width calc * updated packages * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 13:03:11 +02:00
Aliaksandr Valialkin	6e364e19ef	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin	e78f3ac8ac	app/vmselect/bufferedwriter: suppress trivial network errors, which can be generated by remote side ... These errors include `broken pipe` and `reset by peer`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2334	2022-03-18 19:28:02 +02:00
Aliaksandr Valialkin	5d9b9b88b9	app/vmselect/vmui: make vmui-update	2022-03-18 13:18:26 +02:00
Yury Molodov	227d5182af	vmui: update packages (#2264) ... * update package.json * update package-lock.json * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-03 18:49:13 +02:00
Aliaksandr Valialkin	0d47c23a03	app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K ... There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues	2022-02-24 04:05:18 +02:00
Aliaksandr Valialkin	62b46007c5	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin	205d34eae6	app/vmselect: make vmui-update	2022-02-22 21:02:37 +02:00
Yury Molodov	5808774e06	vmui: change logic autocomplete (#2196) ... * fix: change autocomplete display logic * fix: change z-index for labels of input fields * fix: change autocomplete display logic * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:43:55 +02:00
Aliaksandr Valialkin	1d7c877b7b	app/vmui: small fixes ... * Remove unneeded dependency on `numeral` package * Properly parse numbers obtained from /api/v1/query_range according to https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats * Optimize updating processing the received data from /api/v1/query_range * Make smoother zoom on `ctrl+scroll` * Reduce the number of points received from /api/v1/query_range by 2x in order to reduce load on backend	2022-02-14 16:25:46 +02:00
Aliaksandr Valialkin	b1f94f7f0e	app/vmselect/promql: return at most one time series from absent_over_time() in the same way as Prometheus does ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130	2022-02-12 15:45:09 +02:00
Roman Khavronenko	e3adcbec6e	lib/promscrape: support prometheus-like duration in scrape configs (#2169) ... * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:00 +02:00
Aliaksandr Valialkin	af77f449da	app/vmselect/vmui: make vmui-update	2022-02-08 17:26:50 +02:00
Aliaksandr Valialkin	96b7de6736	app/vmselect/promql: clarify comments on why the right side of if and and operators are executed at first	2022-02-03 00:26:14 +02:00
Aliaksandr Valialkin	4b850c2a59	app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values ... Such filters may slow down time series search, so just skip them. This is a follow-up for e7f1ceeb84 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827	2022-02-02 23:40:02 +02:00
Aliaksandr Valialkin	2016a2c899	app/vmselect/promql: properly handle foo or bar queries ... Such queries may miss `bar` results after the commit e7f1ceeb84 because common label filters from `foo` could be mistakenly applied to `bar`.	2022-02-01 17:40:51 +02:00
Aliaksandr Valialkin	4bdd10ab90	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations ... Follow-up for f4989edd96 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin	a8d22e1223	app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first ... PromQL and MetricsQL operators are case-insensitive	2022-01-31 22:06:15 +02:00
Aliaksandr Valialkin	e7f1ceeb84	app/vmselect/promql: optimize queries, which join on _info metrics. ... Automatically add common filters from one side of binary operation to the other side before sending the query to storage subsystem. See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/ and https://www.robustperception.io/exposing-the-software-version-to-prometheus	2022-01-31 19:32:36 +02:00
Aliaksandr Valialkin	7edf8be3bc	app/vmselect/vmui: make vmui-update	2022-01-31 12:36:59 +02:00
Aliaksandr Valialkin	74a2297dcc	app/vmselect/vmui: make vmui-update after e3995572bb	2022-01-25 17:51:46 +02:00
Aliaksandr Valialkin	f4989edd96	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions ... Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:24:44 +02:00
Aliaksandr Valialkin	285bb2bbec	app/vmselect/vmui: make vmui-update	2022-01-24 12:19:49 +02:00
Yury Molodov	ad5059f2d3	vmui: fixed display type switching (#2088) ... * fix: correct switch display type * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 16:56:22 +02:00
Yury Molodov	adbb821eac	vmui: fix time range selector (#2085) ... * fix: add date validate for time range * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 12:02:38 +02:00
Aliaksandr Valialkin	d79a915583	app/vmui: fix last 6 months time range picker ... Previously it was incorrectly selecting 6 minutes instead of 6 months. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960	2022-01-18 23:28:10 +02:00
Aliaksandr Valialkin	672fcba223	app/vmui: properly calculate graph range for y axis ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037	2022-01-18 22:22:24 +02:00
Aliaksandr Valialkin	5a77c86e97	app/vmui: reduce the refresh interval during graph scrolling/zooming from 1 second to 300 milliseconds ... One second feels too laggy, so let's reduce the refresh interval to 300 milliseconds. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064	2022-01-18 21:48:12 +02:00
Yury Molodov	8bdc45ba00	fix: remove buffer period (#2078) ... * fix: remove buffer period * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the implemented feature Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:42:56 +02:00
Yury Molodov	70737ea4ac	vmui: correct url encoding (#2067) ... * fix: correct encode multi-line queries * fix: change autocomplete for correct arrows work * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:31:46 +02:00
Yury Molodov	8e3f9c1fbb	vmui: correct calc axes limits (#2058) ... * fix: correct calc axes limits * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 15:43:41 +02:00
Yury Molodov	fcd33fc409	vmui: change layout (#2054) ... * fix: change query reset * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * feat: move time selector to Header * checkout * fix: remove unused vars * update package-lock.json * fix: correct styles * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 12:44:22 +02:00
Aliaksandr Valialkin	dc7b63a793	app/vmselect/promql: properly keep metric names when optimized path is used for aggregate function calculations ... For example, `sum(rate(...) keep_metric_names) by (__name__)` didn't leave the original metric name because of this issue. This is a follup-up for 1bdc71d917 Udates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-17 15:30:30 +02:00
Aliaksandr Valialkin	1bdc71d917	app/vmselect/promql: implement keep_metric_names modifier for transform and rollup functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-14 04:14:59 +02:00
Aliaksandr Valialkin	f41846d002	app/vmselect/promql: add stale_samples_over_time() function	2022-01-14 01:48:04 +02:00
Aliaksandr Valialkin	c883c15878	app/vmselect/promql: add support for @ modifier ... Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier Extend the support with the following features: * Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()` * Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)` returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348	2022-01-13 22:12:06 +02:00
Aliaksandr Valialkin	9469696e46	app/vmselect/promql: fix limit_offset() test ... The test has been broken in addae7fc6a	2022-01-13 16:59:28 +02:00
Yury Molodov	7d5ed49d23	vmui: switching to Preact (#2053) ... * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * fix: remove unused vars * update package-lock.json * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-11 10:32:17 +02:00
Aliaksandr Valialkin	ceade70d4e	app/vmselect/vmui: run make vmui-update after 89ff7b2465	2022-01-03 21:03:37 +02:00
Yury Molodov	681a800086	vmui: legend fixes (#1995) ... * feat: add a reset query by clicking the logo * feat: add sequence number for query fields * feat: invert behavior on the graph's legend * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-23 12:14:16 +02:00
Roman Khavronenko	543bd0ea0c	vmselect: update /query_exemplars placeholder (#2000) ... Grafana expects `data` in response to be a slice and logs an err if it is not: ``` err="[]v1.ExemplarQueryResult: decode slice: expect [ or n, but found , error found in #0 byte of ...||..., bigger context ...||..." ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-23 11:53:50 +02:00
Yury Molodov	4b40acd964	vmui: add custom start range (#1989) ... * feat: add custom start range * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 20:19:33 +02:00
Yury Molodov	718c352946	vmui: graph fixes (#1982) ... * fix: remove disabling custom step when zooming * feat: add a dynamic calc of the width of the graph * fix: add validate y-axis limits * fix: correct axis limits for value 0 * fix: change logic create time series * fix: change types for tooltip * fix: correct points on the line * fix: change the logic for set graph width * fix: stop checking the period when auto-refresh is enabled * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-20 17:37:02 +02:00
Aliaksandr Valialkin	6d77cc9b08	app/vmselect/vmui: make vmui-update	2021-12-20 13:51:02 +02:00
Aliaksandr Valialkin	974d9c0eee	app/vmselect/promql: follow-up after 177e345d8a ... * Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions. * Simplify their implementation * Mention these functions in docs/CHANGELOG.md	2021-12-20 13:19:44 +02:00
匠心零度	177e345d8a	add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972) ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-12-20 12:32:43 +02:00
Aliaksandr Valialkin	e6d4641bf0	app/vmselect/vmui: make vmui-update	2021-12-17 11:01:09 +02:00
Aliaksandr Valialkin	193331d522	app/vmselect: de-duplicate data exported via /api/v1/export/csv by default ... Previously the exported data wasn't de-duplicated. Now it is possible to export the raw data without deduplication by passing reduce_mem_usage=1 query arg to /api/v1/export/csv See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837	2021-12-17 10:57:39 +02:00
Aliaksandr Valialkin	5efe377a26	app/vmselect/promql: add timestamp_with_name(m[d]) function ... This function works the same as `timestamp()`, but doesn't remove source time series names. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388	2021-12-15 23:37:07 +02:00
Yury Molodov	9baad51004	vmui: introduce application mode (#1949) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * feat: introduce application mode * build vmui * Revert "build vmui" This reverts commit c0e2415550. * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-15 21:33:25 +02:00
Aliaksandr Valialkin	92070cbb67	lib/storage: return dedup interval in milliseconds from GetDedupInterval() ... This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:26:38 +02:00
Aliaksandr Valialkin	1d20a19c7d	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() ... This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:49:12 +02:00
Yury Molodov	49e6a921df	vmui: custom step (#1942) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * docs: document that VMUI now supports overriding of `step` query arg, which is passed to `/api/v1/query_range` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-14 14:51:45 +02:00
Yury Molodov	b5b701d590	vmui: minor fixes (#1936) ... * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document bugfixes Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-13 13:42:37 +02:00
Aliaksandr Valialkin	ce80a0ce5e	app/vmselect/vmui: make vmui-update	2021-12-13 13:32:54 +02:00
Yury Molodov	7c3b6365f0	vmui: add a label for the Query field (#1923) ... * feat: add a label for the Query field * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-12 19:06:39 +02:00
Aliaksandr Valialkin	d1f8915ed1	app/vmselect/promql: preserve the order of time series passed to limit_offset() function ... Previously time series passed to `limit_offset()` were shuffled according to hash for their labels. This was unexpected behaviour for most users. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951	2021-12-12 18:04:58 +02:00
Yury Molodov	c1fd93e8a0	vmui: multiple queries (#1916) ... * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * feat: remove save the last query to local storage * fix: handle an error in a table with subqueries * feat: store display type in URL * Revert "feat: store display type in URL" This reverts commit ccc242c69a. * feat: store display type in URL * refactor: move the time setting to a folder * refactor: move the query configurator to a folder * refactor: move the auth settings to a folder * feat: improve styles * feat: add multi query * update package-lock * feat: add display multiple queries * feat: add limits for multiple queries * update dependencies * feat: add history for multiple queries * feat: add line type to legend * feat: change style for switch * feat: change the logic for axes limits for multiple queries * update package-lock.json * update dependencies * feat: add the filter to legend * wip * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-08 16:40:15 +02:00
Aliaksandr Valialkin	ff15a752c1	app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2021-12-06 17:07:09 +02:00
Aliaksandr Valialkin	732a0cd3e1	app/vmselect/vmui: make vmui-update	2021-12-06 10:19:09 +02:00
Aliaksandr Valialkin	f67427ae61	app/vmselect/vmui: make vmui-update	2021-11-30 01:38:24 +02:00
Yury Molodov	1b58d126c0	vmui: optimize render (#1852) ... * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * update package-lock * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-30 00:56:48 +02:00
Aliaksandr Valialkin	91243ad5cd	app/vmselect/vmui: make vmui-update	2021-11-29 21:55:08 +02:00
Aliaksandr Valialkin	ce31e837eb	app/vmselect/vmui: make vmui-update	2021-11-29 12:22:59 +02:00
Michael Fuller	cf8c171f85	vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838) ... Co-authored-by: Michael Fuller <mfuller@digitalocean.com>	2021-11-22 13:20:42 +03:00
Aliaksandr Valialkin	9bee043ff2	app/vmselect/promql: consistently return zero from deriv(const)	2021-11-17 18:02:05 +02:00
Aliaksandr Valialkin	b900560b83	app/vmselect/promql: add now() function, which returns the current timestamp as a floating-point value in seconds	2021-11-17 16:35:30 +02:00
Aliaksandr Valialkin	f43586c63c	app/vmselect/promql: arrange function names in the code in alphabetical order ... This should simplify code maintenance in the future	2021-11-14 13:55:06 +02:00
Aliaksandr Valialkin	9fa098d8e3	app/vmselect/promql: prevent from incorrect calculations for deriv() over multiple samples with identical timestamps	2021-11-12 13:50:43 +02:00
Aliaksandr Valialkin	e5ac9d8e57	all: consistently return application/json content-type without charset=utf-8 ... The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:04:44 +02:00
Aliaksandr Valialkin	840ac283ef	app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-09 11:41:56 +02:00
Yury Molodov	b78ab88a1c	vmui: migration MUI Core v4 to v5 (#1795) ... * migration MUI Core v4 to v5 * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-08 15:45:59 +02:00
Aliaksandr Valialkin	3419ac1d36	app/vmselect/promql: add duration_over_time(m[d], max_interval) function ... This function calculates the actual lifetime of the time series on the given lookbehdind window `d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-08 13:14:09 +02:00
Yury Molodov	6d1d558c4f	vmui: fix graph reset (#1788) ... * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * feat: change graph legend * update dependencies * update codemirror version * fix: correct update period time after zoom/pan * fix: optimize data processing for the graph * fix: eliminate memory leaks related to mouse events * fix: correct display of straight line * Merge branch 'master' into vmui-fix-reset-graph * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-05 20:56:57 +02:00
Aliaksandr Valialkin	2178335618	app/vmselect: make vmui-update	2021-11-04 12:13:12 +02:00
Aliaksandr Valialkin	27044b84d2	app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778	2021-11-03 16:02:27 +02:00
Aliaksandr Valialkin	43a58bd618	app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names	2021-11-03 13:19:08 +02:00
Yury Molodov	47ee3744f2	vmui: correct migration material-ui (#1758) ... * migration material-ui * fix: rollback popover * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-27 20:41:26 +03:00
Aliaksandr Valialkin	1952ab99aa	all: fix build issues and tests for Apple M1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:06:34 +03:00
Yury Molodov	956cf83e7b	vmui: update dependencies (#1754) ... * update dependencies * update codemirror version * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-26 19:31:20 +03:00
Aliaksandr Valialkin	df8f967040	app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738	2021-10-24 13:31:14 +03:00
Yury Molodov	2b266cb87e	vmui: query history (#1732) ... * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:21:22 +03:00
Yury Molodov	a3e09a57c2	vmui: features (#1711) ... * feat: initial uPlot graph * feat: add zoom/pan for graph * fix: add zoom by ctrl/mac * fix: remove unused code * feat: add toggle cache for fetch * feat: add fix y-axis limits * fix: stop point events while panning * fix: change getting cursor position when scaling * feat: add cursor tooltip to graph * fix: uninstall chart.js * fix: change link for create an issue * fix: set default cache value to true * app/vmalert: follow-up after 0e2486df56 * docs/CHANGELOG.md: document 5416e18007 * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-18 15:16:57 +03:00
Aliaksandr Valialkin	da97e58979	app/vmselect/promql: randomize the static selection of time series returned from limitk() ... Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series. Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.	2021-10-16 21:16:49 +03:00
Aliaksandr Valialkin	cae174b11c	app/vmselect/promql: typo fix in comment: didsn't -> didn't	2021-10-16 13:00:34 +03:00
Aliaksandr Valialkin	a5001b9c20	app/vmselect/promql: add atan2 binary operator, which is going to be added in Prometheus 2.31 ... See https://github.com/prometheus/prometheus/pull/9248	2021-10-11 21:15:53 +03:00
Aliaksandr Valialkin	81c6720392	app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31 ... See https://github.com/prometheus/prometheus/issues/9233	2021-10-11 21:01:33 +03:00
Aliaksandr Valialkin	92b92d4d2c	app/vmselect/promql: consistently return the same set of time series from limitk() function ... This is the expected behaviour by most users.	2021-10-08 19:53:52 +03:00
Aliaksandr Valialkin	0ff8fcac6a	app/vmui: follow-up after 7bfb44113e ... * Run `vmui-update` * Document the changes in README.md and CHANGELOG.md	2021-10-08 15:09:29 +03:00
Aliaksandr Valialkin	2748255c8b	app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow ... The list of functions, which can adjust lookbehind window is more limited than the rest of functions, so it is better from maintainability and readability PoV using the allowlist instead of blocklist.	2021-10-07 13:18:42 +03:00
Aliaksandr Valialkin	c45210a6f9	app/vmselect/promql: return back the behaviour for deriv() function when the lookbehind window doesnt contain enough points ... It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d` contains less than 2 samples in the same way as `rate()` does. This is a follow-up after 3e084be06b .	2021-10-07 12:52:27 +03:00
Roman Khavronenko	3e084be06b	app/vmselect: make predict_linear and deriv compatible with Prometheus (#1681) ... Previously, `predict_linear` returned slightly different results comparing to Prometheus. The change makes linear regression algorithm compatible with Prometheus. `deriv` was excluded from the list of functions which can adjust the time window for the same reasons.	2021-10-07 12:50:49 +03:00
Aliaksandr Valialkin	0e3de5a0cc	app/vmselect/promql: add topk_last and bottomk_last functions	2021-09-30 13:22:52 +03:00
Roman Khavronenko	a31407006c	app/vmselect: fix binary comparison func (#1667) ... The fix makes the binary comparison func to check for NaNs before executing the actual comparison. This prevents VM to return values for non-existing samples for expressions which contain bool comparisons. Please see added test for example.	2021-09-30 12:24:17 +03:00
Roman Khavronenko	344490d89b	app/vmselect: fix testRowsEqual func NaN checks (#1666) ... It appeared, that `testRowsEqual` NaN comparison was incorrect. The fix caused some tests to fail. Please see the change and tests updated. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-30 12:08:47 +03:00
Yury Molodov	893af0a92c	vmui: fixed bug with time range (time zone) (#1661) ... * fix: set date in query string in utc format * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 01:00:44 +03:00
Roman Khavronenko	5dc84bf210	app/vmselect: disable time-window adjustment for min/max_over_time funcs (#1658) ... Adjustment results into discrepancy between Prometheus and VM on time windows smaller than scrape interval. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:43:21 +03:00
Roman Khavronenko	de810031bf	app/vmselect: always return zero for stddev func if there is only one value (#1659) ... The fix will always return zero if received set of items consists of one element only, which also means no deviation. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:38:55 +03:00
Roman Khavronenko	dd536b475c	app/vmselect: return NaN instead of 0 for empty value sets (#1660) ... The change affects `count/stddev/stdvar_over_time` funcs and makes them to return NaN instead of zero when there is no datapoints in a time window. This is needed for improving compatibility with Prometheus. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:37:04 +03:00
Roman Khavronenko	03cd93bf1a	app/vmselect: rm quantile_over_time fast-path optimisations (#1662) ... The removed fast path optimisations weren't consistent with `quantile` function behavior and results into discrepancy. Specifically, results didn't match in cases when: * 0 < phi > 1; * values contain only one element. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:35:14 +03:00
Yury Molodov	a64155d91e	vmui: use Chart.js as default engine for graph (#1634) ... * feat: add Plotly as default engine for graph * fix: remove unused components * feat: use Chart.js as default engine graph * fix: correct styles for loader * feat: add zoom/pan for chart * feat: add height for chart * fix: remove unused code * fix: remove empty units from duration * fix: change debounce for pan to 500ms * fix: add utility for plugins register globally * fix: optimize render graph * feat: add buffer data for zoom * fix: add limits for zoom in/out * fix: change update data while zooming * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-27 22:26:14 +03:00
Aliaksandr Valialkin	2efe0acfc9	app/vmselect/promql: add rollup_scrape_interval(m[d]) function ... It calculates the min, max and avg scrape intervals for m over the given lookbehind window d	2021-09-27 19:21:24 +03:00
Aliaksandr Valialkin	c4c77aa2dd	app/vmselect/promql: follow-up after 526dd93b32 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612	2021-09-27 18:55:39 +03:00
Roman Khavronenko	526dd93b32	app/vmselect: quantile func compatiblity with Prometheus (#1646) ... * app/vmselect: `quantile` func compatiblity with Prometheus The `quantile` func was previously calculated by https://github.com/valyala/histogram package. The result of such calculation was always the closest real value to requested quantile. While in Prometheus implementation interpolation is used. Such difference may result into discrepancy in output between Prometheus and VictoriaMetrics. This commit adds a Prometheus-like `quantile` function. It also used by other functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmselect: `quantile` review fixes * quantile functions were split into multiple to provide different API for already sorted data; * float64sPool is used for reducing allocations. Items in pool may have different sizes, but defining a new pool was complicates due to name collisions; Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-27 18:02:41 +03:00
Aliaksandr Valialkin	8ed95e82c6	app/vmselect/promql: follow-up after 57b3320478	2021-09-24 01:24:18 +03:00
Roman Khavronenko	57b3320478	app/vmselect: make sorting for query result similar to Prometheus (#1647) ... * app/vmselect: make sorting for query result similar to Prometheus Updated sorting allows to get the order of series in result similar or equal to what Prometheus returns. The change is needed for compatibility reasons. * Update app/vmselect/promql/exec_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-24 01:03:12 +03:00
Aliaksandr Valialkin	e564411a62	app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus ... Previously VictoriaMetrics could return unexpected result of the right-hand side operand had multiple time series with the given set of labels mentioned in `on(labels)`. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643	2021-09-24 00:46:25 +03:00
Aliaksandr Valialkin	a22aa0608b	app/vmselect: fix accessing /graphite/* endpoints	2021-09-21 13:56:35 +03:00
Aliaksandr Valialkin	9a3d0c43b5	app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once	2021-09-17 23:35:10 +03:00
Aliaksandr Valialkin	2951dd0a57	app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function ... This function calculates multiple quantiles over the given buckets at once Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-09-17 13:32:39 +03:00
Aliaksandr Valialkin	5a44be0e52	app/vmselect/promql: optimize quantiles() calculation ... Calculate quantiles in one go instead of calculating each quantile individually	2021-09-17 12:33:42 +03:00
f41gh7	9375b60c5f	adds stub for functions api	2021-09-16 13:49:52 +03:00
Aliaksandr Valialkin	e60dfc96ff	app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL	2021-09-16 13:33:53 +03:00
Aliaksandr Valialkin	eca75cc650	app/vmselect/prometheus: make more clear log messages for errors during sending data to remote clients	2021-09-16 12:56:58 +03:00
Aliaksandr Valialkin	2f86d4cf38	app/vmui: follow-up after 777ff75874 ... The commit contains the following changes: - Show vmui when requesting /graph page in order to be compatible with Prometheus datasource in Grafana. - Properly encode query args at vmui url. - Set the number of points on the graph to the number of horizontal pixels divided by 2. Previously it was hardcoded to 30. - Do not save server url to persistent storage at browser, since it should be always obtained from the url. - Run `make vmui-update` for updating vmui embedded into VictoriaMetrics.	2021-09-15 17:40:48 +03:00
Aliaksandr Valialkin	d1a16e0891	app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function	2021-09-13 13:48:06 +03:00
Aliaksandr Valialkin	f4dead529f	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:09:21 +03:00
Aliaksandr Valialkin	ea943911bc	app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600	2021-09-09 19:21:18 +03:00
Aliaksandr Valialkin	5ea689d61b	app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-08-27 18:37:20 +03:00
Aliaksandr Valialkin	bec18e4fe9	app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570	2021-08-27 17:15:31 +03:00
Aliaksandr Valialkin	6c5760db9c	app/vmselect/promql: make fmt after 0078486ea7	2021-08-23 23:06:00 +03:00
Aliaksandr Valialkin	0078486ea7	app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus ... See https://github.com/prometheus/prometheus/pull/8457 for details.	2021-08-23 11:45:51 +03:00
Aliaksandr Valialkin	4f3a5742eb	app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d]) ... This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d	2021-08-21 19:56:14 +03:00
Roman Khavronenko	b5d6a0e499	vmselect: update vm_request_duration_seconds value when request fails (#1537) ... Before, metric `vm_request_duration_seconds` was update only on successful attempts which could be misleading. For example, timeout errors on netstorage request may be not accounted in the metric and won't be visible on dashboards. Using `defer` statement to update the metric after query arguments validation may improve the situation.	2021-08-19 13:58:54 +03:00
Aliaksandr Valialkin	a724229b5d	app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval ... Since most users do not expect the overriding of explicitly set `[d]`.	2021-08-19 10:31:48 +03:00
Aliaksandr Valialkin	cdc372bb98	app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries ... This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:59:02 +03:00
Aliaksandr Valialkin	db1e62495b	app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541	2021-08-17 13:21:21 +03:00
Aliaksandr Valialkin	bd14b0887e	app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability	2021-08-17 11:01:16 +03:00
Aliaksandr Valialkin	113f0a8a07	app/vmselect/promql: drop staleness marks before calling rollupConfig.Do ... This allows dropping staleness marks only once and then calculate multiple rollup functions on the result. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-15 13:21:10 +03:00
Aliaksandr Valialkin	25997a70f1	Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions" ... This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346. It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc	2021-08-15 13:19:16 +03:00
Aliaksandr Valialkin	73d7b568da	app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions ... Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values, which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-14 12:45:57 +03:00
Aliaksandr Valialkin	4401464c22	all: add support for Prometheus staleness markers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin	95dd5a48bb	app/vmselect: make vmui-update after the commit 4ae14df864a7e327955f44941295a286175423b3	2021-08-11 13:41:41 +03:00
Aliaksandr Valialkin	e92fde7945	app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 ... See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097	2021-08-03 16:11:49 +03:00
Nikolay	7bbff7fb86	adds /rules and /alerts api for grafana (#1504) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-08-02 17:28:09 +03:00
Aliaksandr Valialkin	a1911e1330	app/vmselect/netstorage: unpack time series data in mostly local big chunks ... This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples	2021-07-30 12:03:17 +03:00
Aliaksandr Valialkin	74ffaa45d9	app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17	2021-07-30 09:57:53 +03:00
Aliaksandr Valialkin	192dfbfd90	app/vmselect: follow-up for ed95bc9531 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493	2021-07-29 09:53:28 +03:00
arnoldyahad	00af4ff5a4	Add case prometheus/rules for grafana 8 (#1502)	2021-07-29 09:53:27 +03:00
Aliaksandr Valialkin	8ee8660ac4	app/vmselect: follow-up for 626073bca8 ... * Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries * Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes * Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md	2021-07-28 18:00:23 +03:00
Aliaksandr Valialkin	8b7917cd81	all: add go:build lines for Go1.17 ... See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin	95aff47330	app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel	2021-07-26 15:47:30 +03:00
Aliaksandr Valialkin	fcaf152480	app/vmselect: make vmui-update after a91d41f12a	2021-07-26 10:31:11 +03:00
Aliaksandr Valialkin	05672ffc32	app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series ... The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.	2021-07-16 01:44:58 +03:00
Aliaksandr Valialkin	ed10141ff8	app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores	2021-07-16 00:35:23 +03:00
Aliaksandr Valialkin	ca75432e66	app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries ... This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries or queries that take too much cpu time.	2021-07-15 16:44:28 +03:00
Aliaksandr Valialkin	f4e81aef7e	app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series ... This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067	2021-07-15 16:03:28 +03:00
Aliaksandr Valialkin	e6ef97a5ee	app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems	2021-07-15 15:41:58 +03:00
Aliaksandr Valialkin	a925d5a3e1	app/vmselect/promql: duration handling improvements in MetricsQL queries ... - Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600 - Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])	2021-07-12 17:16:41 +03:00
Aliaksandr Valialkin	8c764e88f0	app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-09 17:15:23 +03:00
Aliaksandr Valialkin	c5f0b454f0	app/vmselect: follow-up after aa11ef6d3b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-07 17:43:35 +03:00
tony	e9e35a7d6a	add vmui for vmselect component (#1431) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-07-07 17:33:02 +03:00
Aliaksandr Valialkin	ed944313b0	app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms	2021-07-07 13:25:21 +03:00
Aliaksandr Valialkin	766edbc421	lib/httpserver: print full requestURI in httpserver.Errorf ... This should simplify debugging.	2021-07-07 13:09:40 +03:00
Aliaksandr Valialkin	cdfae0117a	app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect	2021-06-23 14:19:00 +03:00
Aliaksandr Valialkin	83a4db813e	app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration	2021-06-18 19:04:42 +03:00
Roman Khavronenko	fb4f758715	promql: fix increase_pure calculation for cases with stale series (#1381) ... Due to staleness handling, increase_pure were using incorrect previous value during calculation in cases where series disappears for period longer than staleness period and then returns back. The fix suppose to account for a real datapoint value before staleness takes place. The fix should remove unexpected spikes while using `increase_pure` for staled series.	2021-06-15 17:37:19 +03:00
John Belmonte	67b17cdd68	spelling fix: synonym (#1363)	2021-06-10 08:32:52 +03:00
Aliaksandr Valialkin	d0dca62026	app/vmselect/promql: typo fix in the comment	2021-06-09 18:34:31 +03:00
Aliaksandr Valialkin	0842bb9294	app/vmselect/promql: add timezone_offset(tz) function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306	2021-05-20 11:53:09 +03:00
Aliaksandr Valialkin	832651c6c2	app/vmselect: follow up after 8a0678678b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168	2021-05-12 17:18:30 +03:00
Nikolay	8a0678678b	Adds tsdb match filters (#1282) ... * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin	6bc52fe41a	all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com	2021-04-20 20:16:17 +03:00
Aliaksandr Valialkin	544821b719	app/vmselect/promql: fix tests after d3fa0ccabd	2021-04-08 00:18:01 +03:00
Aliaksandr Valialkin	d3fa0ccabd	app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-08 00:09:40 +03:00
Aliaksandr Valialkin	cb12a8f0a8	app/vmselect: return data:null instead of data:[] from /api/v1/query_exemplars, since Grafana throws an error otherwise ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-07 23:34:06 +03:00
Aliaksandr Valialkin	1177dca3da	app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-07 14:16:08 +03:00
Aliaksandr Valialkin	7d23598b33	app/vmselect: return dumb response on /api/v1/query_exemplars request ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-05 23:25:08 +03:00
Aliaksandr Valialkin	7a0b964e8d	app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label)	2021-04-03 22:05:06 +03:00
Aliaksandr Valialkin	3055ab0115	app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value")	2021-04-02 23:55:54 +03:00
Aliaksandr Valialkin	c79e4a2f90	app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users ... Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.	2021-04-02 15:02:08 +03:00
Aliaksandr Valialkin	e7fdea5953	app/vmselect: add -search.maxStatusRequestDuration command-line flag for limiting the duration of requests to /api/v1/status/* and /api/v1/series/count	2021-03-30 21:41:35 +03:00
Aliaksandr Valialkin	7962cf1af8	app/vmselect: prevent from possible incomplete query results after timed out query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 13:35:45 +03:00
Aliaksandr Valialkin	aa81039b42	app/vmselect: log the metric which trigger rollup result cache reset ... This should help finding the source of stale metrics	2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin	0b2a66db30	app/vmselect/promql: do not merge time series during requests to /api/v1/query ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141	2021-03-25 13:56:07 +02:00
Aliaksandr Valialkin	331a6a2015	app/vmselect/graphite: accept and enforce extra_label in all the Graphite APIs	2021-03-23 15:29:16 +02:00
Aliaksandr Valialkin	b521d1d4f2	app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon	2021-03-23 14:16:29 +02:00
Aliaksandr Valialkin	71b72304ae	app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag	2021-03-22 16:45:34 +02:00
Aliaksandr Valialkin	cef010d5f7	app/vmselect/promql: increment key prefix for faster reset for rollup result cache	2021-03-22 11:59:07 +02:00
Aliaksandr Valialkin	b0c956a178	app/vmselect/graphite: follow-up after 529d7be26b	2021-03-18 16:30:20 +02:00
Nikolay	529d7be26b	changes metricsFind api (#1137) ... it should be able mitigate crash if label value contains *,[ or { symbols	2021-03-18 16:12:02 +02:00
Aliaksandr Valialkin	b1713e3fcd	app/vmselect/promql: typo fix after 9666834045	2021-03-17 15:12:11 +02:00
Aliaksandr Valialkin	9666834045	app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function ... This should improve accuracy for the returned buckets	2021-03-17 14:31:41 +02:00
Aliaksandr Valialkin	7bc728bf53	app/vmselect: add vm_index_search_duration_seconds histogram for monitoring the performance of index search	2021-03-17 01:17:41 +02:00
Aliaksandr Valialkin	0ce557951f	app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores	2021-03-16 21:51:31 +02:00
Aliaksandr Valialkin	f0a4157f89	app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time()	2021-03-16 12:24:21 +02:00
Aliaksandr Valialkin	85a95bf60c	all: various fixes in command-line flag descriptions	2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin	923cdb0552	app/vmselect/promql: reduce overhead on scrape interval estimation ... It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.	2021-03-15 20:31:33 +02:00
Aliaksandr Valialkin	82aab87446	app/vmselect/promql: fix tests after 2dae0a2c47	2021-03-15 20:18:59 +02:00
Aliaksandr Valialkin	2dae0a2c47	app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point	2021-03-15 12:36:33 +02:00
Aliaksandr Valialkin	2096c6e464	app/vmselect/prometheus: typo fix after 7c002023d7	2021-03-12 12:19:36 +02:00
Aliaksandr Valialkin	7c002023d7	app/vmselect/prometheus: do not include datapoints with timestamps matching t-d when returning results from /api/v1/query?query=m[d]&time=t as Prometheus does	2021-03-12 12:16:50 +02:00
Aliaksandr Valialkin	fe8b12fbad	app/vmselect/promql: follow up for 433fff0006	2021-03-09 12:48:44 +02:00
Nikolay	433fff0006	duplicate timeseries fix for prometheus_buckets function (#1119) ... * try fix for prometheus_buckets * merge possible end of the bucket collision	2021-03-09 12:26:23 +02:00
Aliaksandr Valialkin	a14053ffa0	app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL	2021-03-04 14:12:07 +02:00
Aliaksandr Valialkin	2b53add6b2	app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration ... This should help debugging slow queries.	2021-02-28 19:40:19 +02:00
Nikolay	186c078fac	adds enforced tag filters into cache key (#1095)	2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin	d86e9b49c4	app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge ... The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets, then the resulting accuracy can suffer.	2021-02-26 22:56:36 +02:00
Aliaksandr Valialkin	98854e5f2b	app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release ... See https://twitter.com/roidelapluie/status/1363428376162295811 The `last_over_time(m[d])` function already exists in MetricsQL.	2021-02-24 17:24:56 +02:00
Aliaksandr Valialkin	f4135b0d14	app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does	2021-02-24 00:42:22 +02:00
Aliaksandr Valialkin	fa03e0d210	app/vmselect/promql: add increase_pure() function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962	2021-02-22 19:14:15 +02:00
Aliaksandr Valialkin	9c70c1f21f	app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics	2021-02-18 22:07:29 +02:00
Aliaksandr Valialkin	c4756f94da	app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations	2021-02-16 16:08:53 +02:00
Aliaksandr Valialkin	5a401225c7	app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key ... This should prevent from a memory allocation and a string copy.	2021-02-16 15:43:10 +02:00
Aliaksandr Valialkin	4e39bf148c	vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 ... The new version switches from log-linear histograms to log-based histograms, which provide up to 3.6 times better accuracy.	2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin	acdb401585	app/vmselect/prometheus: treat match query arg in the same way as match[] query arg	2021-02-11 15:02:21 +02:00
Aliaksandr Valialkin	2d33230793	app/vmselect/promql: properly make copies of EvalConfig	2021-02-11 12:41:15 +02:00
Aliaksandr Valialkin	7a3a9421f3	app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2 ... This should prevent from data races if the underlying functions modify EvalConfig contents.	2021-02-10 23:05:55 +02:00
Aliaksandr Valialkin	04faea8b45	app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel ... This should reduce query execution times.	2021-02-10 22:59:39 +02:00
Aliaksandr Valialkin	553016ea99	lib/storage: disable composite index usage when querying old data	2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin	c7dccebaef	lib/storage: optimize search by label filters matching big number of time series	2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin	803a00102a	app/{vminsert,vmselect}: accept requests to paths with /graphite and /prometheus prefixes ... This should improve compatibility with path prefixes from VictoriaMetrics cluster. See https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#url-format	2021-02-04 20:01:18 +02:00
Aliaksandr Valialkin	4d43ab0875	app/vmselect: typo fix when stripping url path prefixes	2021-02-04 19:29:25 +02:00
Aliaksandr Valialkin	8629fd8a72	app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax	2021-02-03 20:36:01 +02:00
Aliaksandr Valialkin	d16f22f3a1	app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards ... Example: foo{bar{x,yz},a[b-c],*de}	2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin	157c02622b	app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax	2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin	b43b498fd8	app/vmselect: add ability to pass extra_label=<label>=<value> query arg to Prometheus Querying API ... This enforced `{label="value"}` label filter to the query. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021	2021-02-01 18:04:17 +02:00
Nikolay	195341a7cf	Graphite vmalert wip (#112) ... * init implementation for graphite alerts * adds graphite support for vmalert * small fix * changes vmalert graphite api with type * updates tests * small fix * fixes graphite parse * Fixes graphite from time	2021-02-01 15:05:32 +02:00
Aliaksandr Valialkin	c164a8d231	app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag ... This should reduce incorrect usage and assumptions for this flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020	2021-01-22 13:00:10 +02:00
Aliaksandr Valialkin	8749c2dd92	app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time	2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin	011c5da785	app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()	2021-01-18 17:30:26 +02:00
Aliaksandr Valialkin	1051d8aa2d	app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992	2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin	acf1a2c72b	app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990	2021-01-13 10:59:42 +02:00
Aliaksandr Valialkin	1837f2f7d3	app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d	2021-01-12 16:12:12 +02:00
Aliaksandr Valialkin	c86286ec1d	app/vmselect/promql: do not ajdust offset value provided in the query ... Previously it could be modified in order to improve response cache hit ratio. This is unneeded, since cache hit ratio should remain good because the query time range should be already aligned to multiple of `step` values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976	2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin	4b7105a65b	app/vmselect: sync query stats handling with cluster version	2020-12-27 13:00:29 +02:00
Aliaksandr Valialkin	df0309eae0	app/vmselect/promql: simplify defer call for querystats.RegisterQuery	2020-12-27 12:06:04 +02:00
Aliaksandr Valialkin	ad4e6a9283	app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K ... This should reduce memory usage in constrained environments	2020-12-25 17:40:47 +02:00
Aliaksandr Valialkin	59183f66d0	app/vmselect: refactor /api/v1/stats/top_queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907	2020-12-25 16:44:29 +02:00
Nikolay	86630350bf	Adds query stats handler (#945) ... * Adds query stat handler, for query and query_range api, victoriametrics tracks query execution time, stats are expored at /api/v1/status/queries endpoint with topN param https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907 * fixed query stats bugs * improves queryStats tracker * improves query stat * small fix * fix tests * added more tests * fixes 386 tests * naming fixes * adds drop for outdated records	2020-12-25 16:42:05 +02:00
Aliaksandr Valialkin	88ac4dfc07	app/vmselect: properly parse negative combined offsets such as -1h2m3s ... Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.	2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin	11fa458e39	app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100	2020-12-15 13:40:46 +02:00
Aliaksandr Valialkin	5ebfc275e6	app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested	2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin	ba8ac08739	app/vmselect/graphite: properly handle wildcards and charsets inside curly braces ... For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952	2020-12-11 17:24:12 +02:00
Aliaksandr Valialkin	4146fc4668	all: properly handle CPU limits set on the host system/container ... This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin	1906f841c9	app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect	2020-12-08 15:16:17 +02:00
Aliaksandr Valialkin	007dbf273d	app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942	2020-12-07 01:08:52 +02:00
Aliaksandr Valialkin	e2e8ef86d9	app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N	2020-12-05 12:30:46 +02:00
Aliaksandr Valialkin	66379cc69f	app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936	2020-12-03 21:47:36 +02:00
Aliaksandr Valialkin	44a34a0f5f	app/vmselect/promql: make fmt	2020-12-02 21:33:35 +02:00
Aliaksandr Valialkin	1982505c2b	app/vmselect/promql: return nan from minute(m) when m equals to nan ... This aligns VictoriaMetrics behaviour with Prometheus behaviour. The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 20:16:58 +02:00
Aliaksandr Valialkin	9d87496b50	app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. ... This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.	2020-12-02 13:12:50 +02:00
Aliaksandr Valialkin	91a4c279cc	app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true ... This aligns MetricsQL behavior to Prometheus' one. The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 12:09:34 +02:00
Aliaksandr Valialkin	700bda8e2e	app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does	2020-12-02 00:28:26 +02:00
Aliaksandr Valialkin	efdc3c71af	app/vmselect/searchutils: return elapsed time in Deadline.String() output ... This should improve debuggability for error messages containing Deadline.String() output	2020-12-01 00:15:18 +02:00
Aliaksandr Valialkin	a724dde90a	app/vmselect: protect /tags/delSeries with -deleteAuthKey in the same way as /api/v1/admin/tsdb/delete_series	2020-11-23 15:35:59 +02:00
Aliaksandr Valialkin	f0c207fae2	app/vmselect: add /tags/delSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb	2020-11-23 15:27:21 +02:00
Aliaksandr Valialkin	d3794eb994	app/{vminsert,vmselect}: move /tags/tagSeries and /tags/tagMultiSeries api from vminsert to vmselect ... This is needed for consistency, since all the `/tags*` api handlers are located in vmselect.	2020-11-23 12:33:19 +02:00
Aliaksandr Valialkin	f765985947	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin	2859a452d4	app/vmselect: add remoteAddr to slow query log in order to improve debuggability ... This will simplify identifying the client that sends slow queries to VictoriaMetrics.	2020-11-18 20:38:32 +02:00
Aliaksandr Valialkin	266788be14	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place ... This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin	cf18df367d	app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name	2020-11-16 15:52:16 +02:00
Aliaksandr Valialkin	2c67232565	app/vmselect/graphite: add /tags/autoComplete/values handler from Graphite Tags API	2020-11-16 15:29:35 +02:00
Aliaksandr Valialkin	86f99c6b55	app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support	2020-11-16 14:50:05 +02:00
Aliaksandr Valialkin	3c1434118e	app/vmselect/prometheus: return __name__ label if match[] query to /api/v1/labels matches at least a single time series	2020-11-16 13:54:34 +02:00
Aliaksandr Valialkin	27a417bcd3	app/vmselect/prometheus: improve performance for /api/v1/labels and /api/v1/label/<labelName>/values on time ranges exceeding one day when match[] query arg is set	2020-11-16 13:51:59 +02:00
Aliaksandr Valialkin	6fa806f1ca	app/vmselect/prometheus: fix deadlock in /api/v1/series on a time range exceeding one day	2020-11-16 13:30:47 +02:00
Aliaksandr Valialkin	465923b181	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin	414cd39659	app/vmselect/graphite: apply filter then limit	2020-11-16 04:09:14 +02:00
Aliaksandr Valialkin	d100341394	app/vmselect/graphite: add /tags/<tag_name> handler for Graphite Tags API	2020-11-16 03:42:25 +02:00
Aliaksandr Valialkin	6251762787	app/vmselect/graphite: add /tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 03:00:01 +02:00
Aliaksandr Valialkin	d9d01f976b	app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points ... Do not take into account spare data point value if the next point will is located too far from the current point. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 15:23:44 +02:00
Aliaksandr Valialkin	1f19c167a4	app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() ... This should remove unexpected spikes at the end of gaps. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 14:59:24 +02:00
Aliaksandr Valialkin	47a038401b	all: consistently return text-based HTTP responses with charset=utf-8 ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:35:41 +02:00
Aliaksandr Valialkin	348edd92fe	app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters	2020-11-11 12:39:07 +02:00
Aliaksandr Valialkin	47390d8947	app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2020-11-10 14:51:44 +02:00
Aliaksandr Valialkin	ba4a2c8bca	app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum	2020-11-10 01:18:08 +02:00
Aliaksandr Valialkin	dd6bfa50e9	app/vmselect/promql: code cleanup after 43823addea	2020-11-06 01:30:50 +02:00
n4mine	43823addea	app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883)	2020-11-06 01:29:24 +02:00
Aliaksandr Valialkin	368b69b4c4	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin	b378cd6ed8	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin	caeb74f068	app/vmselect: reduce memory usage when query touches big number of time series	2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin	5e75c389e6	app/vmselect/promql: allow dropping trailing sample only for default_rollup function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850	2020-11-02 02:10:59 +02:00
Aliaksandr Valialkin	cb71af216a	app/vmselect/promql: go fmt	2020-11-02 00:15:29 +02:00
Aliaksandr Valialkin	daacbc7e34	app/vmselect/promql: do not drop trailing datapoints for instant queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-11-02 00:12:37 +02:00
Aliaksandr Valialkin	c539494b36	app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	28353e48ca	app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series ... Now trailing data points are additionally dropped for time series with a single raw sample Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-17 10:44:34 +03:00
Aliaksandr Valialkin	a2b9476897	app/vmselect/promql: return a single time series at max from absent() function like Prometheus does	2020-10-13 15:56:04 +03:00
Aliaksandr Valialkin	9aa3b65766	app/vmselect/promql: improve time series staleness detection ... This should prevent from double counting for time series at the time when it changes label. The most common case is in K8S, which changes pod uid label with each new deployment. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-13 12:19:57 +03:00
Aliaksandr Valialkin	d8af290947	app/vmselect/promql: fix mode_over_time calculations ... Previously `mode_over_time` could return garbage due to improper shuffling of input data points.	2020-10-13 11:58:25 +03:00
Aliaksandr Valialkin	1e27420243	app/vmselect/prometheus: fix golangci-lint warning	2020-10-13 09:36:11 +03:00
Aliaksandr Valialkin	4f16a964e3	app/vmselect: add ability to export data in CSV format via /api/v1/export/csv	2020-10-12 20:08:17 +03:00
Aliaksandr Valialkin	762c967855	app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning ... Functions are: * keep_last_value * keep_next_value * interpolate * running_min * running_max * running_avg * range_min * range_max * range_avg * range_first * range_last * range_quantile * smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:47:06 +03:00
Aliaksandr Valialkin	45f7cdc532	Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" ... This reverts commit ac45082216. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric names: * ceil * floor * round Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:40:34 +03:00
Aliaksandr Valialkin	a94825b169	Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" ... This reverts commit bb61a4769b. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * clamp_min * clamp_max Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:38:27 +03:00
Aliaksandr Valialkin	f7d28bddbf	Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" ... This reverts commit e5202a4eae. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * max_over_time * min_over_time * avg_over_time * quantile_over_time * geomean_over_time * mode_over_time * holt_winters * predict_linear Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:35:18 +03:00
Aliaksandr Valialkin	e9f2e2cbc9	app/vmselect/promql: add missing label filters to binary operands before query execution ... This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization See also https://github.com/cortexproject/cortex/issues/3253	2020-10-07 21:15:09 +03:00
Aliaksandr Valialkin	59c26feefa	app/vmselect/promql: fill gaps on graphs for range_* and running_* functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806	2020-10-02 13:59:45 +03:00
Aliaksandr Valialkin	f7636b0342	app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output	2020-09-29 18:03:26 +03:00
Aliaksandr Valialkin	96ee276e6e	app/vmselect/prometheus: check for errors returned from bufferedwriter.Write ... This makes `make errcheck` happy	2020-09-29 11:37:01 +03:00
Aliaksandr Valialkin	6fdfc67620	app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.* ... In this case only node must be returned with stripped dot in the end of id as carbonapi does	2020-09-29 11:01:59 +03:00
Aliaksandr Valialkin	1b3efccb24	app/vmselect: stop /api/v1/export/* execution if client disconnects	2020-09-27 23:53:13 +03:00
Aliaksandr Valialkin	95688cbfc5	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin	82973f8ae7	Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" ... This reverts commit bab6a15ae0. Reason for revert: the `fetchData` arg is used in cluster branch. Leaving this arg in master branch makes smaller the diff with cluster branch.	2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin	bab6a15ae0	lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock ... This arg became unused after 23bdc1f107	2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin	23bdc1f107	app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls	2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin	c584aece38	app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784	2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin	27500d7d4c	app/vmselect/prometheus: code cleanup after 3ba507000c	2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin	3ba507000c	app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin	bed25e3c24	app/vmselect/netstorage: properly pre-allocate space for sbs	2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin	09b0f7c202	app/vmselect/netstorage: release search resources on timeout errors ... Previously these resources weren't released, which could lead to resource leaks.	2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin	3b1e3a03e0	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin	00b5145c69	app/vmselect/searchutils: fixed tests after 2eb72e09ab	2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin	2eb72e09ab	app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:40 +03:00
Aliaksandr Valialkin	98d1cd0971	app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API	2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin	9f79bcf64a	app/vmselect: improve description for -search.maxQueryDuration	2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin	1fec47a289	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin	8c3d7c1a59	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin	0e533d1a9c	app/vmselect/promql: support composite durations like Prometheus 2.21 does ... The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 23:39:13 +03:00
Aliaksandr Valialkin	a2f647d142	app/vmselect/prometheus: typo fix in the description for -search.latencyOffset command-line flag	2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin	f0005c3007	app/vmselect: move Deadline from netstorage to searchutils ... This removes dependency on netstorage from searchutils.	2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin	2114179e19	app/vmselect: substitute inf values at smooth_exponential with the previous values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin	204ec415b4	app/vmselect: skip infinite values when calculating smooth_exponential ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin	8a8b5a73d3	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin	f6bc608e86	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin	62919eaf7e	app/vmselect/promql: go fmt	2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin	e6da63dffe	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin	8e85b56737	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin	c0343a661b	app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3	2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin	f4e7e5fb90	app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions ... These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin	e706e59d49	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin	147c35ebd4	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin	7d89fafe1a	app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)	2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin	e969ef2639	app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin	8adba82c02	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores ... This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin	f3d33e23c9	app/vmstorage: improve error logging when the request times out	2020-08-10 13:23:26 +03:00
Aliaksandr Valialkin	716754fae6	app/vmselect/promql: properly handle -n^m like Prometheus does ... `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin	bb61a4769b	app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin	ac45082216	app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin	e5202a4eae	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus ... Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin	68e4f40a72	app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does ... This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin	bc8381613d	app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names	2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin	3f85c06b65	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block ... This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin	d20c2156e4	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin	5d0c37bec0	app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration	2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin	e6eee2bebf	app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...)	2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin	509d12643b	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin	f0c678c41b	app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin	328b52e5ff	app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window ... Just divide the data point value by the window duration in this case.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	700737c181	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now ... See e00cfc854d for the approach used now.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	2f735f112d	app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	79c30cf4cb	app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query	2020-07-28 15:13:08 +03:00
Aliaksandr Valialkin	0da202023b	app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp ... This aligns `group()` behaviour to Prometheus	2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin	894dcb7c1c	app/vmselect/promql: improve further the accuracy of buckets_limit() function ... The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin	edb1eca6f1	app/vmselect/promql: avoid dropping inf bucket in buckets_limit ... The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin	97b6f5d223	app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets	2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin	bb161497cf	app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function ... Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin	994fa2f3bf	app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k ... This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin	e151c5c644	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin	3107c633e3	app/vmselect/promql: typo fix after 3e557c9861	2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin	3e557c9861	app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration ... Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin	b1f6843bd0	app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time ... This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin	039c9d2441	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin	2a45871823	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin	461481fbdf	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery	2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin	e79de9774b	app/vmselect: typo fix after 34563916f7	2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin	34563916f7	app/vmselect: reduce memory usage when querying big number of time series with long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin	4f526cc816	app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin	dfb113f175	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries ... This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.	2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin	caa2952aa6	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin	e00cfc854d	app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value ... This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin	b9c8f6bf34	app/vmselect: log the total available memory for concurrent requests on not enough memory errors ... This should simplify root cause analysis	2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin	5a4675c528	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range ... This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin	ecb1b2564a	app/vmselect/promql: add mode() aggregate function	2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin	b35cb293f5	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin	1c641037e8	app/vmselect/promql: add mode_over_time(m[d]) function ... See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin	6b5ad535ae	app/vmselect/promql: optimize group(rollup(m)) calculations	2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin	8949d65ad1	app/vmselect/promql: check that any() doesn't touch metric name	2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin	aa5d88055d	app/vmselect/promql: add group() aggregate function to MetricsQL ... This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin	df01836818	app/vmselect/promql: keep all labels for time series from any() call	2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin	e4e1cd1de2	app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage	2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin	25f80d320b	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time ... Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin	55d83e777d	app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9	2020-07-13 21:40:38 +03:00
faceair	1033dc7e2a	fix empty response template (#617)	2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin	9c350bc20d	app/vmselect/promql: add missing tests for ifnot binary operation	2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin	256fd9a87e	app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation	2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin	2d9b3ad5b3	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin	742da690f4	app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575	2020-07-08 18:55:38 +03:00
DexterZhang	99f54e44ff	feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598)	2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin	acf828a759	app/vmselect/prometheus: small fixes on top of 8bb762124a	2020-07-05 18:17:06 +03:00
faceair	8bb762124a	fix adjust last points avoid influence earlier value (#606)	2020-07-05 17:56:54 +03:00
Aliaksandr Valialkin	5d3db3ff7c	app/vmselect: add interpolate function for filling gaps with linearly interpolated values ... See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin	84a37098ed	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention ... VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention if `-denyQueriesOutsideRetention` command-line flag is set.	2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin	7c2c8b2981	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin	05d754d7bb	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series ... This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:38:39 +03:00
nicbaz	774f7ca1c1	vmselect: fix label_replace when mismatch (#579) ... As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the **MetricsQL** `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin	7209d58fbd	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series ... Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:46:15 +03:00
Aliaksandr Valialkin	7532dbcdf5	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin	88e1b7d144	app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin	dbd8beccfa	app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions ... These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:07:48 +03:00
Aliaksandr Valialkin	70ea4e28a7	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin	538fdfe133	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin	f52769f6ee	app/vmselect/promql: fix outilersk calculations	2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin	a441cdd1d9	app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin	0afd48d2ee	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin	7882a0dbbf	app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint	2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin	faf92a0965	app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one	2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin	cc311e20fe	app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y	2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin	574289c3fb	app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation	2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin	57407cca83	app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag ... Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin	4e4f57b121	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin	17d96e4503	app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs vs false-positive not enough memory errors	2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin	b4afe562c1	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin	fcf57f9883	app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks ... This should minimize the number of disk seeks when reading data from temporary file.	2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin	a596aec82c	app/vmselect: fix description for -search.resetCacheAuthKey	2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin	364db13c9c	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin	9ebc937685	app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin	fe57d46687	app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( ... Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin	f41e6a7bd9	app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query	2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin	071fdf5518	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin	6f7f64f757	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin	f9b24d4899	app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions	2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin	318326c309	lib/storage: properly handle {label=~"foo|"} filters as Prometheus does ... Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin	0c485f14d1	app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries	2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin	2ebf7d86ff	app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()	2020-03-29 21:50:28 +03:00
Aliaksandr Valialkin	3c7c71a49c	app/vmselect: adjust label_map() handling for corner cases ... The following corner cases now supported: * label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label` * label_map(q, "label", "foo", "") - removes `label="foo"` from series All the unmatched labels are kept unchanged.	2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin	fdc2a9d1d7	app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin	8023ad7dbd	app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model	2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin	e17702fada	app/vmselect: add optional max_rows_per_line query arg to /api/v1/export ... This arg allows limiting the number of data points that may be exported on a single line.	2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin	c5f894b361	Makefile: add build and test rules with enabled race detector. These rules have -race suffix ... Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin	7d178a40bd	app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter ... This should reduce query time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin	5701b2f7bb	app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin	18af31a4c2	all: properly split vm_deduplicated_samples_total among cluster components ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin	ae51300973	app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin	e65ec88779	app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-23 18:03:57 +02:00
Yaroslav	a6d0645539	fix rollupOpen(), rollupHigh(), rollupLow() functions (#328)	2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin	2f55cabaa4	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin	ec9bf39b5b	app/vmselect: add -search.cacheTimestampOffset command-line flag ... This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin	539139391c	app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:05 +02:00
Aliaksandr Valialkin	7b1c7051a3	app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions ... This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin	a6c6a2debc	app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8	2020-02-05 21:22:19 +02:00