VictoriaMetrics/lib/storage
Artem Fetishev 5f89374cc8
lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064)
### Describe Your Changes

Currently it the metricID list is empty it won't be mashalled and as the
result won't be put into the tagFiltersToMetricIDsCache which causes the
cache misses for the corresponding tagFilters. In some setups this
causes severe search speed detradation (see #7009).

The empty metric IDs was covered before but then was accidentally
removed in 6c21439.

This PR restores the coverage of this case.

A new unit test can be used as a proof that empty metricID lists are not
added to the cache (just remove the fix in index_db.go and run the test
to see the result)

Also a benchmark has been added to see the implications of the
compression.

```
user@laptop:~/p/github.com/rtm0/VictoriaMetrics/01/src$ go test ./lib/storage/ -run=NONE -bench BenchmarkMarshalUnmarshalMetricIDs --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-0-12             3237240               363.5 ns/op               0 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1-12             2831049               451.8 ns/op               0.4706 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10-12            1152764              1009 ns/op                 1.667 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100-12            297055              3998 ns/op                 5.755 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000-12            31172             34566 ns/op                 8.484 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000-12            4900            289659 ns/op                 9.416 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100000-12            447           2341173 ns/op                 9.456 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000000-12            42          24926928 ns/op                 9.468 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000000-12            5         204098872 ns/op                 9.467 compression-rate
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  15.018s
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-20 17:38:56 +02:00
..
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:48:44 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
block_stream_reader_test.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:25:49 +02:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_reader.go lib/storage: avoid superflouos copy of block header data 2024-02-23 01:11:31 +02:00
block_stream_writer_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
block_stream_writer.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
block_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-26 09:44:40 +02:00
block.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup.go docs: mention staleNaN handling during deduplication 2024-01-16 20:11:45 +02:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:34:36 -07:00
index_db_test.go lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:38:56 +02:00
index_db_timing_test.go lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:38:56 +02:00
index_db.go lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:38:56 +02:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
merge_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
merge_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
merge.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
metaindex_row_test.go
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:04 +03:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
metric_name.go lib/storage: change default value for maxLabelValueLen to 1024 (#6313) 2024-05-22 21:55:21 +02:00
part_header.go lib/storage: add ability to use downsampling for the given series filter (#733) 2024-04-03 02:38:37 +03:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_search_timing_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-26 09:44:40 +02:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:31:39 -08:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
partition_search_test.go lib/storage: do not keep rawRows buffer across flush() calls 2024-02-23 01:06:09 +02:00
partition_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
partition_test.go lib/storage: stop exposing vm_merge_need_free_disk_space metric 2023-09-25 17:00:14 +02:00
partition.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:43:00 +02:00
raw_block.go
raw_row.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
search_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:55:07 +02:00
search.go lib/storage: fix SearchQuery.Unmarshal() after 32193b6059 2024-05-14 01:39:01 +02:00
storage_test.go lib/storage: adds metrics that count records that failed to insert 2024-09-06 18:13:48 +02:00
storage_timing_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:55:07 +02:00
storage.go lib/storage: simplify indexDB.doExtDB() usage by removing the returned value 2024-09-20 17:37:03 +02:00
table_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table_search_timing_test.go lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency 2024-02-24 00:51:03 +02:00
table_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
table_test.go flagutil: Make .Msecs private (#4906) 2023-09-03 10:37:57 +02:00
table_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:43:00 +02:00
tag_filters_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
tag_filters_timing_test.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-03 03:00:45 +03:00
tag_filters.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
time_test.go
time.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
tsid_test.go
tsid.go