VictoriaMetrics/lib
Artem Fetishev 5f89374cc8
lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064)
### Describe Your Changes

Currently it the metricID list is empty it won't be mashalled and as the
result won't be put into the tagFiltersToMetricIDsCache which causes the
cache misses for the corresponding tagFilters. In some setups this
causes severe search speed detradation (see #7009).

The empty metric IDs was covered before but then was accidentally
removed in 6c21439.

This PR restores the coverage of this case.

A new unit test can be used as a proof that empty metricID lists are not
added to the cache (just remove the fix in index_db.go and run the test
to see the result)

Also a benchmark has been added to see the implications of the
compression.

```
user@laptop:~/p/github.com/rtm0/VictoriaMetrics/01/src$ go test ./lib/storage/ -run=NONE -bench BenchmarkMarshalUnmarshalMetricIDs --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-0-12             3237240               363.5 ns/op               0 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1-12             2831049               451.8 ns/op               0.4706 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10-12            1152764              1009 ns/op                 1.667 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100-12            297055              3998 ns/op                 5.755 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000-12            31172             34566 ns/op                 8.484 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000-12            4900            289659 ns/op                 9.416 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100000-12            447           2341173 ns/op                 9.456 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000000-12            42          24926928 ns/op                 9.468 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000000-12            5         204098872 ns/op                 9.467 compression-rate
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  15.018s
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-20 17:38:56 +02:00
..
appmetrics
auth
awsapi
backup lib/backup/s3remote: add retryer configuration (#6747) 2024-08-07 16:59:23 +02:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
bufferedwriter
buildinfo
bytesutil lib/bytesutil: smooth buffer growth rate (#6761) 2024-08-07 16:59:23 +02:00
cgroup lib/cgroup: remove SetGOGC() function 2024-02-05 12:13:08 +02:00
consts
decimal lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
encoding lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
envflag
envtemplate
envutil testing: allow disabling fsync to make tests run faster (#6871) 2024-08-30 11:18:21 +02:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:24 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
filestream vlinsert: added opentelemetry logs support 2024-09-03 20:24:01 +02:00
flagutil lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag() 2024-07-16 01:08:41 +02:00
formatutil
fs lib/fs: properly call windows APIs (#6998) 2024-09-13 13:19:04 +02:00
handshake
htmlcomponents
httpserver app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert 2024-09-06 23:00:58 +02:00
httputils lib/httputils: parse URL before creating HTTP transport (#6820) 2024-08-16 11:34:49 +02:00
influxutils app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749) 2024-08-05 09:45:32 +02:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:57:49 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 17:02:05 +02:00
logger lib/logger: increase default value of -loggerMaxArgLen cmd-line fla… (#7008) 2024-09-19 15:48:09 +02:00
logstorage lib/logstorage: make sure that getCommonTokens returns common tokens in the original order of tokens inside tokenSets arg 2024-09-19 16:00:21 +02:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
memory
mergeset lib/mergeset: fix typos in comments 2024-08-07 16:59:22 +02:00
metricsql
netutil lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
persistentqueue app/vmagent/remotewrite: follow-up for 87fd400dfc 2024-07-13 02:30:10 +02:00
procutil
promauth lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
prompb Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
prompbmarshal Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
promrelabel lib/promrelabel: follow-up for 8958cecad6 2024-08-27 15:44:07 +02:00
promscrape lib/discovery/azure: fix host check in next link in Azure SD (#6915) 2024-09-05 16:58:10 +02:00
promutils all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
protoparser follow-up after 01430a155c 2024-09-04 15:39:55 +02:00
proxy lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
pushmetrics
querytracer make go vet happy 2024-08-19 21:41:44 +02:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-04-03 02:41:11 +03:00
regexutil lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
snapshot lib/httputils: parse URL before creating HTTP transport (#6820) 2024-08-16 11:34:49 +02:00
storage lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:38:56 +02:00
streamaggr tests: fix slice init length (#6897) 2024-08-30 11:18:21 +02:00
stringsutil all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
syncwg
tenantmetrics
timerpool
timeutil
uint64set lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now 2024-07-16 00:00:46 +02:00
vmselectapi lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00