41 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	92826b0b4a	app/vmselect/promql: apply SLO-like optimization to all the count_*_over_time() functions ... This is a follow-up for 41a0fdaf39	2023-11-01 09:58:16 +01:00
Aliaksandr Valialkin	41a0fdaf39	app/vmselect/promql: optimize repeated SLI-like instant queries with lookbehind windows >= 1d ... Repeated instant queries with long lookbehind windows, which contain one of the following rollup functions, are optimized via partial result caching: - sum_over_time() - count_over_time() - avg_over_time() - increase() - rate() The basic idea of optimization is to calculate rf(m[d] @ t) as rf(m[offset] @ t) + rf(m[d] @ (t-offset)) - rf(m[offset] @ (t-d)) where rf(m[d] @ (t-offset)) is cached query result, which was calculated previously The offset may be in the range of up to 1 hour.	2023-10-31 19:25:23 +01:00
Nikolay	1f91f22b5f	app/vmselect: reduce lock contention for heavy aggregation requests (#5119) ... reduce lock contention for heavy aggregation requests previously lock contetion may happen on machine with big number of CPU due to enabled string interning. sync.Map was a choke point for all aggregation requests. Now instead of interning, new string is created. It may increase CPU and memory usage for some cases. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087	2023-10-10 13:45:20 +02:00
Aliaksandr Valialkin	140e7b6b74	all: replace atomic.Value with atomic.Pointer[T] ... This eliminates the need in .(*T) casting for results obtained from Load() Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map, because map is already a pointer type.	2023-07-19 17:42:06 -07:00
Aliaksandr Valialkin	30425ca81a	lib/fs: rename WriteFileAtomically to MustWriteAtomic ... Callers of this function log the returned error and exit. So let's just log the error with the given filepath and the call stack inside the function itself and then exit. This simplifies the code at callers' place while leaves the same level of debuggability in case of errors.	2023-04-13 22:41:15 -07:00
Aliaksandr Valialkin	e8225d7d6b	app/vmselect/promql: prevent from cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0 ... The issue has been introduced in 0af9e2b693	2023-03-12 19:09:39 -07:00
Aliaksandr Valialkin	30ed33fae0	app/vmselect/promql: pre-allocate memory for values to be merged in mergeTimeseries() ... This should reduce the number of memory re-allocations	2023-01-09 22:51:17 -08:00
Aliaksandr Valialkin	645c24dc5f	app/vmselect/promql: consistently intern series names obtained from marshalMetricNameSorted ... This reduces memory allocations when the returned series names are used as map keys later	2023-01-09 22:45:40 -08:00
Aliaksandr Valialkin	2f3ddd4884	app/vmselect/promql: avoid memory allocations and copying from source timeseries to the returned result at timeseriesToResult()	2023-01-09 22:38:59 -08:00
Aliaksandr Valialkin	c4265322f4	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin	a10647e0bf	app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"}	2022-10-23 12:13:47 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin	3e2dd85f7d	all: readability improvements for query traces ... - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:20:33 +03:00
Aliaksandr Valialkin	a963b2a0aa	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	12ac255dae	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin	41958ed5dd	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin	61c7f6beae	app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2406	2022-04-11 12:34:43 +03:00
Aliaksandr Valialkin	62b46007c5	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin	ff15a752c1	app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2021-12-06 17:07:09 +02:00
Aliaksandr Valialkin	bec18e4fe9	app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570	2021-08-27 17:15:31 +03:00
Aliaksandr Valialkin	aa81039b42	app/vmselect: log the metric which trigger rollup result cache reset ... This should help finding the source of stale metrics	2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin	cef010d5f7	app/vmselect/promql: increment key prefix for faster reset for rollup result cache	2021-03-22 11:59:07 +02:00
Nikolay	186c078fac	adds enforced tag filters into cache key (#1095)	2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin	5ebfc275e6	app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested	2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin	f0c678c41b	app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin	0afd48d2ee	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin	4e4f57b121	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin	2f55cabaa4	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin	ec9bf39b5b	app/vmselect: add -search.cacheTimestampOffset command-line flag ... This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin	680080887d	all: consistently log durations in seconds with millisecond precision ... This should improve logs readability	2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin	3e09d38f29	app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time ... Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.	2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin	1925ee038d	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:03:59 +02:00
Mike Poindexter	bec62e4e43	Split Extended PromQL parsing to a separate library	2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin	f4a38f7fb1	app/vmselect/promql: fix panic on -search.disableCache ... Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.	2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin	9ecb994671	app/vmselect/promql: store compressed results in the cache ... This should increase rollup results cache capacity.	2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin	09fc6e22e5	all: use workingsetcache instead of fastcache ... This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin	a336bb4e22	app/vmselect/promql: reduce RAM usage for aggregates over big number of time series ... Calculate incremental aggregates for `aggr(metric_selector)` function instead of keeping all the time series matching the given `metric_selector` in memory.	2019-07-10 13:04:39 +03:00
Aliaksandr Valialkin	1fe6d784d8	all: consistency renaming: bytesSize -> sizeBytes	2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin	53ea90865d	app/vmselect/promql: add -search.disableCache flag for disabling response caching ... This may be useful for data back-filling, when the response caching could interfere badly with newly added data points with timestamps in the past.	2019-06-04 17:30:45 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00