Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,842 Commits 114 Branches 748 Tags 509 MiB
b133de1e37
Commit Graph

162 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
b133de1e37 lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin
ce10bdc82a lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin
1e4a64844d lib/storage: properly account the number of loops spent when matching for or suffixes 
This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin
402a8ca710 lib/storage: do not populate MetricID->MetricName cache during data ingestion 
This cache isn't needed during data ingestion, so there is no need in spending RAM on it.

This reduces RAM usage on data ingestion path by 30%
 2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin
2839055513 lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss 2021-05-13 09:01:05 +03:00
Aliaksandr Valialkin
008ae25b3a lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate 
These functions are non-trivial, while their code has minimal differences.
It is better from maintainability PoV to merge these functions into a single function.
 2021-05-12 18:01:08 +03:00
Nikolay
be87be34a4 Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin
4e59cf4380 lib/storage: properly apply time range when matching an empty filter 
It must match all the time series on the given time range.
Previously it was matched to all the time series without the restriction on the given time range.
 2021-05-11 01:09:35 +03:00
Aliaksandr Valialkin
326cf83eb4 lib/storage: remove dead code after the commit 3ccf7ea20c 2021-05-08 20:15:59 +03:00
Aliaksandr Valialkin
72c41323fa lib/storage: code clarification: remove caching the found metricName in searchMetricName 2021-04-13 10:20:35 +03:00
Aliaksandr Valialkin
59ccc43e3a lib/storage: properly handle big time ranges passed to /api/v1/labels and /api/v1/label/<labelName>/values 
It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch
 2021-04-07 13:33:10 +03:00
Aliaksandr Valialkin
f971fe86cd lib/storage: tune loopsCountPerMetricNameMatch according to production workload 2021-03-25 13:48:17 +02:00
Aliaksandr Valialkin
40e47935e7 lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache 2021-03-22 23:02:59 +02:00
Aliaksandr Valialkin
1618b0ca6d lib/storage: tune loopsCountPerMetricNameMatch 2021-03-22 12:57:54 +02:00
Aliaksandr Valialkin
b859fe7879 lib/storage: faster move heavy filters to the end of list 2021-03-17 15:11:56 +02:00
Aliaksandr Valialkin
41fe707bec lib/storage: limit loops count in order to reduce max CPU usage during filter search 2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
e2a0c8bd72 lib/storage: do not modify filterLoopsCount stats with loopsCount stats 
Such a modification can result in incorrect filter sorting later
 2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
727ded9d4e lib/storage: time series search optimization according to production workload profiling 
Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down
the function by multiple times when it finds big number of metricIDs (tens of millions).
 2021-03-16 20:08:43 +02:00
Aliaksandr Valialkin
f4a44d6c0d lib/storage: further tuning for time series search 2021-03-16 18:47:29 +02:00
Aliaksandr Valialkin
d074326970 app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate 2021-03-15 22:40:28 +02:00
Aliaksandr Valialkin
fc902734d9 lib/storage: further tuning for time series selector code 2021-03-15 20:32:37 +02:00
Aliaksandr Valialkin
1c26020080 lib/storage: tune per-day index search 2021-03-15 13:36:36 +02:00
Aliaksandr Valialkin
b2732575f7 lib/storage: further tune filters sorting logic 2021-03-12 00:51:35 +02:00
John Belmonte
edf39aa225 spelling fix: adjacent (#1115) 2021-03-09 09:19:16 +02:00
Aliaksandr Valialkin
c4a0bd5eac lib/storage: go fmt 2021-03-08 11:59:31 +02:00
Aliaksandr Valialkin
c76a904bb0 lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow 
The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload
 2021-03-07 21:17:48 +02:00
Aliaksandr Valialkin
06676c8feb lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache 2021-02-23 15:50:08 +02:00
faceair
b1409a7413 lib/storage: correct tagfilter match cost (#1079) 2021-02-22 21:54:37 +02:00
Aliaksandr Valialkin
587132555f lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin
bd3bcdc43c lib/storage: do not re-calculate stats for heavy tag filters 
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
628b8eb55e lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. 
This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters
 2021-02-18 14:01:18 +02:00
Aliaksandr Valialkin
fd41f070db lib/storage: sort tag filters by the number of loops they need for the execution 
This metric should work better than the filter execution duration, since it cannot be distorted
by concurrently running queries.
 2021-02-18 12:52:29 +02:00
Aliaksandr Valialkin
2005d8212a lib/storage: return back filter arg to getMetricIDsForTagFilter function 
The filter arg has been removed in the commit c7ee2fabb8
because it was preventing from caching the number of matching time series per each tf.

Now the cache contains duration for tf execution, so the filter shouldn't break such caching.
 2021-02-17 19:33:15 +02:00
Aliaksandr Valialkin
83da939947 app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics 2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin
bbc287ea6a lib/storage: tag filters sorting... 2021-02-17 18:56:12 +02:00
Aliaksandr Valialkin
f5e841c1e9 lib/storage: further tune tag filters sorting 2021-02-17 17:28:36 +02:00
Aliaksandr Valialkin
9d1f14d94c lib/storage: tune the logic for sorting tag filters according the their exeuction times 2021-02-17 15:02:19 +02:00
Aliaksandr Valialkin
1ab7a8dfd5 lib/storage: more tuning for tag filters sorting according the time they take 2021-02-16 21:27:27 +02:00
Aliaksandr Valialkin
55952f8f2e lib/storage: tune sorting for tag filters 2021-02-16 13:07:42 +02:00
Aliaksandr Valialkin
3eae03a337 lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs 2021-02-15 16:39:52 +02:00
Aliaksandr Valialkin
6f3bbf21b8 lib/storage: sort tag filters by actual execution time instead of by the number of matching time series 
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
 2021-02-15 00:19:46 +02:00
Aliaksandr Valialkin
4e645a5fd3 lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin
08f21d8761 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin
b27288f1b0 lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin
4262c2f7c2 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin
148422bcba lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
17d5a03f6e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin
fa0ef143b1 lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
4b930b9ffe app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:17:19 +02:00