Nikolay 30b32583f4 lib/storage: pre-create timeseries before indexDB rotation (#4652) ... * lib/storage: pre-create timeseries before indexDB rotation during an hour before indexDB rotation start creating records at the next indexDB it must improve performance during switch for the next indexDB and remove ingestion issues. Since there is no need for creation new index records for timeseries already ingested into current indexDB https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 * lib/storage: further work on indexdb rotation optimization - Document the change at docs/CHAGNELOG.md - Move back various caches from indexDB to Storage. This makes the change less intrusive. The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID) entries for both the current and the next indexDB. - Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function. This improves code readability and maintainability a bit. - Rewrite and simplify the code responsible for calculating the next retention timestamp. Add various tests for corner cases of this code. - Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called. It is OK to add indexdb entries on demand in this function. This simplifies the code. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 * docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>		2023-07-22 15:23:14 -07:00
..
block_header_test.go	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:48:44 +02:00
block_header.go	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block	2022-09-06 13:07:49 +03:00
block_stream_merger.go	lib/storage: skip blocks outside the configured retention during search	2022-10-24 02:56:13 +03:00
block_stream_reader_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
block_stream_reader_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
block_stream_reader.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
block_stream_writer_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
block_stream_writer.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
block_test.go	lib/storage: use deterministic random generator in tests	2023-01-23 20:12:32 -08:00
block.go	lib/storage: validate timestamps in the block only if they use encoding, which needs validation	2022-10-21 00:54:37 +03:00
dedup_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:18:49 -08:00
dedup_timing_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:18:49 -08:00
dedup.go	app,lib: fix typos in comments (#3804)	2023-02-13 09:32:35 -08:00
filenames.go	lib/storage: consistently use OS-independent separator in file paths	2023-03-25 14:34:36 -07:00
index_db_test.go	lib/storage: pre-create timeseries before indexDB rotation (#4652)	2023-07-22 15:23:14 -07:00
index_db_timing_test.go	lib/storage: move series registration in caches from createAllIndexesForMetricName into a separate function - putSeriesToCache	2023-07-13 23:17:14 -07:00
index_db.go	lib/storage: pre-create timeseries before indexDB rotation (#4652)	2023-07-22 15:23:14 -07:00
inmemory_part_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
inmemory_part_timing_test.go	lib/storage: use deterministic random generator in tests	2023-01-23 20:12:32 -08:00
inmemory_part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
merge_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
merge_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
merge.go	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:20 -08:00
metaindex_row_test.go
metaindex_row.go	all: subsitute ioutil.ReadAll with io.ReadAll	2022-08-22 00:16:04 +03:00
metric_name_test.go	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
metric_name.go	app/vmselect/promql: add the ability to copy all the labels from one side of group_left()/group_right() operation	2023-07-17 16:58:30 -07:00
part_header.go	lib/storage: fix a bug, which prevents from reading pre-v1.90.0 parts	2023-04-14 22:33:29 -07:00
part_search_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
part_search_timing_test.go	lib/storage: skip missing tsids in the current block header by using binary search	2022-12-14 22:07:55 -08:00
part_search.go	lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid	2022-12-19 10:31:39 -08:00
part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
partition_search_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
partition_search.go	lib/storage: fixes finalDedup for backfilled data (#3737)	2023-02-01 09:57:02 -08:00
partition_test.go	lib/storage: remove unused code (#4050)	2023-04-02 21:23:24 -07:00
partition.go	docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix	2023-07-06 22:42:02 -07:00
raw_block.go
raw_row.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:47:20 -07:00
search_test.go	lib/storage: replace OpenStorage() with MustOpenStorage()	2023-04-14 23:04:42 -07:00
search.go	app,lib: fix typos in comments (#3804)	2023-02-13 09:32:35 -08:00
storage_test.go	lib/storage: pre-create timeseries before indexDB rotation (#4652)	2023-07-22 15:23:14 -07:00
storage_timing_test.go	lib/storage: replace OpenStorage() with MustOpenStorage()	2023-04-14 23:04:42 -07:00
storage.go	lib/storage: pre-create timeseries before indexDB rotation (#4652)	2023-07-22 15:23:14 -07:00
table_search_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
table_search_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
table_search.go	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg	2022-10-24 01:32:56 +03:00
table_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
table_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
table.go	lib/storage: do not create flock.lock files at partition directories, since it is created at the Storage level	2023-07-06 17:26:37 -07:00
tag_filters_test.go	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:43 +03:00
tag_filters_timing_test.go	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:43 +03:00
tag_filters.go	lib/storage: optimize matching speed for non-trivial regexp filters	2022-10-01 12:07:18 +03:00
time_test.go
time.go	app,lib: fix typos in comments (#3804)	2023-02-13 09:32:35 -08:00
tsid_test.go
tsid.go