Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-25 20:00:06 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
4,673 Commits 113 Branches 748 Tags 707 MiB
90649de0c4
Commit Graph

109 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
92630c1ab4
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system 
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
 2022-07-25 09:18:44 +03:00
Aliaksandr Valialkin
159c2e15e3
app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples 2022-07-12 12:31:38 +03:00
Aliaksandr Valialkin
8429d4af5a
app/vmselect/netstorage: add mergeSortBlocks benchmark for the worstcase 2022-07-12 12:31:36 +03:00
Aliaksandr Valialkin
cd09f583fe
app/vmselect/netstorage: add benchmarks for mergeSortBlocks 
This is a follow-up for 743ff84863
 2022-07-11 12:54:48 +03:00
Aliaksandr Valialkin
743ff84863
app/vmselect/netstorage: optimize mergeSortBlocks function 
- Use binary search instead of linear scan when locating the run of smallest timestamps
  in blocks with intersected time ranges. This should improve performance
  when merging blocks with big number of samples

- Skip samples with duplicate timestamps. This should increase query performance
  in cluster version of VictoriaMetrics with the enabled replication.
 2022-07-09 00:34:42 +03:00
Aliaksandr Valialkin
77cbbacfdb
lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics 
This reduces the number of args to vmselectapi calls
 2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
f435924ab3
lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call 2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
e1b8059086
lib/vmselectapi: rename deleteMetrics to more correct deleteSeries 2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
a60e03b3a7
lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() 
This improves the API consistency
 2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin
a14188dd8e
app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload 
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792
 2022-06-28 20:18:13 +03:00
Aliaksandr Valialkin
a350d1e81c
lib/storage: return marshaled metric names from SearchMetricNames 
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.

While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.

Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
 2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin
e578549b8a
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day 2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin
ba514284f1
lib/storage: add querytracer to more contexts 
querytracer has been added to the following storage.Storage methods:
- RegisterMetricNames
- DeleteMetrics
- SearchTagValueSuffixes
- SearchGraphitePaths
 2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
9a314106ca
app/vmselect/netstorage: remove Get prefix from netstorage functions 
This makes these function names more consistent with the server side
 2022-06-27 00:45:05 +03:00
Aliaksandr Valialkin
ec7963208d
app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb 
This allows filling the seriesCountByFocusLabelValue list in the /api/v1/status/tsdb response
with label values for the specified focusLabel, which contain the highest number of time series.

TODO: add this to Cardinality explorer at VMUI - https://docs.victoriametrics.com/#cardinality-explorer
 2022-06-14 18:36:54 +03:00
Aliaksandr Valialkin
374beb350e
app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them 2022-06-12 04:32:13 +03:00
Aliaksandr Valialkin
89b778902b
app/vmselect: add optional limit query arg to /api/v1/labels and /api/v1/label_values endpoints 
This arg allows limiting the number of sample values returned from these APIs
 2022-06-10 09:50:33 +03:00
Aliaksandr Valialkin
2bcb960f17
all: improve query tracing coverage for indexdb search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
 2022-06-09 20:07:07 +03:00
Aliaksandr Valialkin
12ac255dae
lib/querytracer: make it easier to use by passing trace context message to New and NewChild 
The context message can be extended by calling Donef.
If there is no need to extend the message, then just call Done.
 2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin
41958ed5dd
all: add initial support for query tracing 
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
 2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin
6e364e19ef
app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs 2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin
4bdd10ab90
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 
Follow-up for f4989edd96

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin
f4989edd96
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions 
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().

Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.

Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-25 15:24:44 +02:00
Aliaksandr Valialkin
92070cbb67
lib/storage: return dedup interval in milliseconds from GetDedupInterval() 
This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.
 2021-12-15 13:26:38 +02:00
Aliaksandr Valialkin
1d20a19c7d
lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() 
This improves the code readability and debuggability, since the output of these functions
stops depending on global state.
 2021-12-14 20:49:12 +02:00
Aliaksandr Valialkin
8ed95e82c6
app/vmselect/promql: follow-up after 57b3320478 2021-09-24 01:24:18 +03:00
Aliaksandr Valialkin
f4dead529f lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2021-09-09 21:09:21 +03:00
Aliaksandr Valialkin
a1911e1330 app/vmselect/netstorage: unpack time series data in mostly local big chunks 
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
 2021-07-30 12:03:17 +03:00
Aliaksandr Valialkin
74ffaa45d9 app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17 2021-07-30 09:57:53 +03:00
Aliaksandr Valialkin
8ee8660ac4 app/vmselect: follow-up for 626073bca8 
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
 2021-07-28 18:00:23 +03:00
Aliaksandr Valialkin
95aff47330 app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel 2021-07-26 15:47:30 +03:00
Aliaksandr Valialkin
ed10141ff8 app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores 2021-07-16 00:35:23 +03:00
Aliaksandr Valialkin
f4e81aef7e app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series 
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
 2021-07-15 16:03:28 +03:00
Aliaksandr Valialkin
e6ef97a5ee app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems 2021-07-15 15:41:58 +03:00
Aliaksandr Valialkin
832651c6c2 app/vmselect: follow up after 8a0678678b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168
 2021-05-12 17:18:30 +03:00
Nikolay
8a0678678b
Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin
7962cf1af8 app/vmselect: prevent from possible incomplete query results after timed out query 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2021-03-30 13:35:45 +03:00
Aliaksandr Valialkin
7bc728bf53 app/vmselect: add vm_index_search_duration_seconds histogram for monitoring the performance of index search 2021-03-17 01:17:41 +02:00
Aliaksandr Valialkin
0ce557951f app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores 2021-03-16 21:51:31 +02:00
Aliaksandr Valialkin
c4756f94da app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations 2021-02-16 16:08:53 +02:00
Aliaksandr Valialkin
5a401225c7 app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key 
This should prevent from a memory allocation and a string copy.
 2021-02-16 15:43:10 +02:00
Aliaksandr Valialkin
553016ea99 lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
c7dccebaef lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
157c02622b app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
007dbf273d app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942
 2020-12-07 01:08:52 +02:00
Aliaksandr Valialkin
f765985947 lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin
cf18df367d app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name 2020-11-16 15:52:16 +02:00
Aliaksandr Valialkin
86f99c6b55 app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support
 2020-11-16 14:50:05 +02:00
Aliaksandr Valialkin
465923b181 app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
 2020-11-16 12:53:13 +02:00