Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-19 15:06:25 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,522 Commits 114 Branches 748 Tags 602 MiB
455bf50a91
Commit Graph

33 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
bc8381613d app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names 2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
3f85c06b65 app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block 
This should improve query performance on systems with big number of CPU cores (16 and more)
 2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin
d20c2156e4 app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans 2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin
039c9d2441 lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
461481fbdf app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery 2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin
e79de9774b app/vmselect: typo fix after 34563916f7 2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin
34563916f7 app/vmselect: reduce memory usage when querying big number of time series with long labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
 2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin
dfb113f175 app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries 
This should slightly reduce load on GC when processing queries that touch big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.
 2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin
caa2952aa6 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin
e4e1cd1de2 app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage 2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin
84a37098ed app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 
VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention
if `-denyQueriesOutsideRetention` command-line flag is set.
 2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
05d754d7bb app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series 
This should reduce GC pressure when processing time series with big number of rows
 2020-06-24 19:38:39 +03:00
Aliaksandr Valialkin
7209d58fbd app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series 
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
 2020-06-23 22:46:15 +03:00
Aliaksandr Valialkin
b4afe562c1 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
fcf57f9883 app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks 
This should minimize the number of disk seeks when reading data from temporary file.
 2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin
364db13c9c app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
318326c309 lib/storage: properly handle {label=~"foo|"} filters as Prometheus does 
Such filters must match all the time series with `label="foo"` plus all the time series without `label`

Previously only time series with `label="foo"` were matched.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
18af31a4c2 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
c3d86eef96 all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
efc7ad88ec app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin
96ff8d9adb app/vmselect: add ability to pass match[], start and end to /api/v1/labels 
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.

See https://github.com/prometheus/prometheus/issues/6178 for more details.
 2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin
20812008a7 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
4bb88843cf app/vmselect: add vm_per_query_{rows,series}_processed_count histograms 2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
c1d3705be0 app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock 
This allows re-using the destination buffer for marshaling in the outer loop.
 2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin
b7ee2e7af2 app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series 2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin
603a79b357 app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries 2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin
47e4b50112 app/vmselect: optimize /api/v1/series by skipping storage data 
Fetch and process only time series metainfo.
 2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin
afcc7fb167 app/vmselect/netstorage: improve error message when reading data blocks from storage 
Mention the block number in the error. This should simplify troubleshooting in this code.
 2019-07-28 12:12:35 +03:00
Aliaksandr Valialkin
ab88890523 app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores 
This may reduce response times for aggregation over big number of time series
with small step between output data points.
 2019-07-12 15:52:22 +03:00
Aliaksandr Valialkin
cbe692f0e2 app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values 2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00