Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-25 20:00:06 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,508 Commits 113 Branches 748 Tags 707 MiB
7bc728bf53
Commit Graph

147 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
ccfb0ae2d3 lib/storage: limit loops count in order to reduce max CPU usage during filter search 2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
576a80b3d9 lib/storage: do not modify filterLoopsCount stats with loopsCount stats 
Such a modification can result in incorrect filter sorting later
 2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
fd86a7dc1d lib/storage: time series search optimization according to production workload profiling 
Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down
the function by multiple times when it finds big number of metricIDs (tens of millions).
 2021-03-16 20:01:43 +02:00
Aliaksandr Valialkin
e36fbfae5b lib/storage: further tuning for time series search 2021-03-16 18:46:22 +02:00
Aliaksandr Valialkin
dd7e82c34f app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate 2021-03-15 22:38:50 +02:00
Aliaksandr Valialkin
9c77e34ef9 lib/storage: further tuning for time series selector code 2021-03-15 20:31:34 +02:00
Aliaksandr Valialkin
3ccf7ea20c lib/storage: tune per-day index search 2021-03-15 13:31:55 +02:00
Aliaksandr Valialkin
f669531506 lib/storage: further tune filters sorting logic 2021-03-12 00:53:04 +02:00
John Belmonte
364fdf4a56
spelling fix: adjacent (#1115) 2021-03-09 09:18:19 +02:00
Aliaksandr Valialkin
345980f78f lib/storage: go fmt 2021-03-08 12:03:31 +02:00
Aliaksandr Valialkin
18fe0ff14b lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow 
The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload
 2021-03-07 21:12:35 +02:00
Aliaksandr Valialkin
2c44178645 lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache 2021-02-23 15:47:19 +02:00
faceair
15d61c4879
lib/storage: correct tagfilter match cost (#1079) 2021-02-22 21:46:56 +02:00
Aliaksandr Valialkin
636c55b526 lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:06:47 +02:00
Aliaksandr Valialkin
388cdb1980 lib/storage: do not re-calculate stats for heavy tag filters 
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
 2021-02-21 21:39:01 +02:00
Aliaksandr Valialkin
e540c02014 lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. 
This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters
 2021-02-18 13:58:18 +02:00
Aliaksandr Valialkin
711f8a5b8d lib/storage: sort tag filters by the number of loops they need for the execution 
This metric should work better than the filter execution duration, since it cannot be distorted
by concurrently running queries.
 2021-02-18 12:47:38 +02:00
Aliaksandr Valialkin
faad6f84a4 lib/storage: return back filter arg to getMetricIDsForTagFilter function 
The filter arg has been removed in the commit c7ee2fabb8
because it was preventing from caching the number of matching time series per each tf.

Now the cache contains duration for tf execution, so the filter shouldn't break such caching.
 2021-02-17 19:33:22 +02:00
Aliaksandr Valialkin
d4849561ef app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics 2021-02-17 19:13:38 +02:00
Aliaksandr Valialkin
63fc140624 lib/storage: tag filters sorting... 2021-02-17 17:55:29 +02:00
Aliaksandr Valialkin
74424b55ee lib/storage: further tune tag filters sorting 2021-02-17 17:28:15 +02:00
Aliaksandr Valialkin
442fcfec5a lib/storage: tune the logic for sorting tag filters according the their exeuction times 2021-02-17 15:00:08 +02:00
Aliaksandr Valialkin
d61f7b7279 lib/storage: more tuning for tag filters sorting according the time they take 2021-02-16 21:22:23 +02:00
Aliaksandr Valialkin
ca191696fe lib/storage: tune sorting for tag filters 2021-02-16 13:04:49 +02:00
Aliaksandr Valialkin
ecf132933e lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs 2021-02-15 16:34:23 +02:00
Aliaksandr Valialkin
71c417427c lib/storage: sort tag filters by actual execution time instead of by the number of matching time series 
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
 2021-02-15 00:18:13 +02:00
Aliaksandr Valialkin
0e26b7168a lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:41:04 +02:00
Aliaksandr Valialkin
b51c23dc5b lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 18:12:25 +02:00
Aliaksandr Valialkin
c7ee2fabb8 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 18:12:20 +02:00
Aliaksandr Valialkin
cdecf83ce5 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:08 +02:00
Aliaksandr Valialkin
553016ea99 lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
fcb7655d1e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:45 +02:00
Aliaksandr Valialkin
c7dccebaef lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
a4140de9e6 lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:47:50 +02:00
Aliaksandr Valialkin
157c02622b app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
4aaee33860 lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page 2020-11-16 00:44:35 +02:00
Aliaksandr Valialkin
b378cd6ed8 app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set 2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
fe289331dd lib/storage: remove obsolete code 2020-11-02 19:11:59 +02:00
faceair
d2960a20e0
evaluate the execution cost of all tag filters (#824) 
* evaluate the execution cost of all tag filters

* fix suffixes typo
 2020-10-17 00:46:55 +03:00
Aliaksandr Valialkin
764dc2499f lib/storage: code cleanup after 10f2eedee0 
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.

While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
 2020-10-01 19:06:23 +03:00
Aliaksandr Valialkin
10f2eedee0 lib/storage: imrpove cache effectiveness for time series ids matching the given filters 
Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day.
This should reduce CPU usage in the following cases:

* when querying recently added data with small churn rate for time series
* when querying historical data
 2020-10-01 14:38:25 +03:00
Aliaksandr Valialkin
a69234ed18 lib/storage: code prettifying after be5e1222f3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
 2020-09-22 00:36:45 +03:00
faceair
be5e1222f3
add filter to getMetricIDs (#783) 
* add getMetricIDs filter

* check nil filter before use
 2020-09-22 00:33:43 +03:00
Aliaksandr Valialkin
f6bc608e86 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
9043a509a3 lib/storage: properly check timeouts and pace limits 
Previously they were checked on every iteration for small number of iterations
 2020-08-07 08:40:37 +03:00
Aliaksandr Valialkin
8e44fba76d lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x 2020-08-06 18:45:55 +03:00
Aliaksandr Valialkin
54ef2d8112 lib/storage: slightly reduce code difference between single-node and cluster versions 2020-07-24 00:31:16 +03:00
Aliaksandr Valialkin
039c9d2441 lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
d3442b40b2 lib/uint64set: optimize adding items to the set via Set.AddMulti 2020-07-21 20:56:59 +03:00