Zakhar Bessarab 837d0d136d lib/mergeset: add sparse indexdb cache (#7269) ... Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182 - add a separate index cache for searches which might read through large amounts of random entries. Primary use-case for this is retention and downsampling filters, when applying filters background merge needs to fetch large amount of random entries which pollutes an index cache. Using different caches allows to reduce effect on memory usage and cache efficiency of the main cache while still having high cache hit rate. A separate cache size is 5% of allowed memory. - reduce size of indexdb/dataBlocks cache in order to free memory for new sparse cache. Reduced size by 5% and moved this to a separate cache. - add a separate metricName search which does not cache metric names - this is needed in order to allow disabling metric name caching when applying downsampling/retention filters. Applying filters during background merge accesses random entries, this fills up cache and does not provide an actual improvement due to random access nature. Merge performance and memory usage stats before and after the change: - before  - after  --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>		2024-10-24 15:21:17 +02:00
..
block_header_test.go
block_header.go	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit	2024-05-14 01:23:54 +02:00
block_stream_merger.go	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:20:37 +02:00
block_stream_reader_test.go
block_stream_reader_timing_test.go
block_stream_reader.go	lib/storage: avoid superflouos copy of block header data	2024-02-22 20:21:14 +02:00
block_stream_writer_timing_test.go	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types	2024-02-24 00:15:26 +02:00
block_stream_writer.go	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types	2024-02-24 00:15:26 +02:00
block_test.go
block.go	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit	2024-05-14 01:23:54 +02:00
dedup_test.go
dedup_timing_test.go
dedup.go
filenames.go
index_db_test.go	lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064)	2024-09-20 17:21:53 +02:00
index_db_timing_test.go	lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064)	2024-09-20 17:21:53 +02:00
index_db.go	lib/mergeset: add sparse indexdb cache (#7269)	2024-10-24 15:21:17 +02:00
inmemory_part_test.go
inmemory_part_timing_test.go
inmemory_part.go
merge_test.go	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types	2024-02-24 00:15:26 +02:00
merge_timing_test.go	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types	2024-02-24 00:15:26 +02:00
merge.go	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:20:37 +02:00
metaindex_row_test.go
metaindex_row.go
metric_name_test.go
metric_name.go	lib/storage: change default value for maxLabelValueLen to 1024 (#6313)	2024-05-22 21:53:53 +02:00
part_header.go	lib/storage: add ability to use downsampling for the given series filter (#733)	2024-03-30 04:12:23 +02:00
part_search_test.go
part_search_timing_test.go
part_search.go
part.go
partition_search_test.go	lib/storage: do not keep rawRows buffer across flush() calls	2024-02-22 17:22:26 +02:00
partition_search.go	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:20:37 +02:00
partition_test.go
partition.go	Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)"	2024-07-25 14:32:09 +02:00
raw_block.go
raw_row.go	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types	2024-02-24 00:15:26 +02:00
search_test.go	Fix inconsistent error handling in Storage.AddRows() (#6583)	2024-07-17 12:07:14 +02:00
search.go	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit	2024-05-14 01:23:54 +02:00
storage_test.go	lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267)	2024-10-23 11:48:21 +02:00
storage_timing_test.go	Fix inconsistent error handling in Storage.AddRows() (#6583)	2024-07-17 12:07:14 +02:00
storage.go	(app|lib)/vmstorage: do not increment vm_rows_ignored_total on NaNs (#7166)	2024-10-02 12:37:27 +02:00
table_search_test.go
table_search_timing_test.go	lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency	2024-02-24 00:53:30 +02:00
table_search.go	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:20:37 +02:00
table_test.go
table_timing_test.go
table.go	Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)"	2024-07-25 14:32:09 +02:00
tag_filters_test.go	lib/logstorage: work-in-progress	2024-05-24 03:06:55 +02:00
tag_filters_timing_test.go	all: fix golangci-lint(revive) warnings after 0c0ed61ce7	2024-04-02 23:16:29 +03:00
tag_filters.go	lib/logstorage: work-in-progress	2024-05-24 03:06:55 +02:00
time_test.go
time.go
tsid_test.go
tsid.go