Aliaksandr Valialkin
7aa5b48508
app/vmselect: reduce memory usage when querying big number of time series with long labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin
bd4299fafe
app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries
...
This should slightly reduce load on GC when processing queries that touch big number of time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile
2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin
5bb4fe1ba4
app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query
2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
14dc426b45
app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage
2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin
4cb3e7595c
app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention
2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
12b87b2088
app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series
...
This should reduce GC pressure when processing time series with big number of rows
2020-06-24 19:37:35 +03:00
Aliaksandr Valialkin
0fdbe5de25
app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series
...
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin
e933cbac16
lib/storage: postpone reading data from blocks during search
...
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.
This improves single-node VM performance on heavy queries by up to 2x.
2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin
23a310cc68
app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks
...
This should minimize the number of disk seeks when reading data from temporary file.
2020-04-26 16:46:17 +03:00
Aliaksandr Valialkin
f9526809e5
app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
cf9aee4ec3
all: properly split vm_deduplicated_samples_total among cluster components
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin
e3adc095bd
all: add -dedup.minScrapeInterval command-line flag for data de-duplication
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin
e127173984
app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors
2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
97de50dd4c
app/vmselect/netstorage: improve error message when reading data size in readBytes
2019-12-24 14:40:14 +02:00
Aliaksandr Valialkin
6a185b7809
app/vmselect: add ability to pass match[], start and end to /api/v1/labels
...
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.
See https://github.com/prometheus/prometheus/issues/6178 for more details.
2019-12-15 00:20:43 +02:00
Aliaksandr Valialkin
f8298c7f13
app/vmselect: add vm_per_query_{rows,series}_processed_count histograms
2019-11-23 13:23:03 +02:00
Aliaksandr Valialkin
946ca438a6
app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock
...
This also allows marshaling outside lock, thus reducing the amount of work under the lock.
2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin
e92e39eddf
app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series
2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin
56dff57f77
app/vmselect/netstorage: reduce memory usage when fetching big number of data blocks from vmstorage
...
Dump data blocks directly to temporary file instead of buffering them in RAM
2019-09-28 12:21:57 +03:00
Aliaksandr Valialkin
b101064f8b
all: report the number of bytes read on io.ReadFull error
...
This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin
880b1d80b1
app/vmselect: optimize /api/v1/series by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
5b8526e925
app/vmselect/netstorage: improve error message when reading data blocks from storage
...
Mention the block number in the error. This should simplify troubleshooting in this code.
2019-07-28 12:17:33 +03:00
Aliaksandr Valialkin
6875fb411a
app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores
...
This may reduce response times for aggregation over big number of time series
with small step between output data points.
2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin
945894e049
app/vmselect: properly handle empty label (aka __name__) in LabelEntries handler
2019-06-10 19:55:02 +03:00
Aliaksandr Valialkin
75a0acf72d
app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values
2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin
a06b7f7f84
app/vmselect/netstorage: remove spammy error message when certain vmstorage nodes are unavailable during query execution
...
The amount of partial responses may be tracked by `vm_partial_search_results_total` metric.
2019-06-03 17:09:50 +03:00
Aliaksandr Valialkin
24578b4bb1
all: open-sourcing cluster version
2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
