Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
9f440b1013
VictoriaMetrics/lib/storage
History
Nikolay 30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652) 
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
..
block_header_test.go
block_header.go lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block 2022-09-06 13:07:49 +03:00
block_stream_merger.go lib/storage: skip blocks outside the configured retention during search 2022-10-24 02:56:13 +03:00
block_stream_reader_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_reader.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_writer_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_writer.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
block.go lib/storage: validate timestamps in the block only if they use encoding, which needs validation 2022-10-21 00:54:37 +03:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:34:36 -07:00
index_db_test.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:23:14 -07:00
index_db_timing_test.go lib/storage: move series registration in caches from createAllIndexesForMetricName into a separate function - putSeriesToCache 2023-07-13 23:17:14 -07:00
index_db.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:23:14 -07:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
merge_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
merge_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
merge.go lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention 2022-12-03 22:14:20 -08:00
metaindex_row_test.go
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:04 +03:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
metric_name.go app/vmselect/promql: add the ability to copy all the labels from one side of group_left()/group_right() operation 2023-07-17 16:58:30 -07:00
part_header.go lib/storage: fix a bug, which prevents from reading pre-v1.90.0 parts 2023-04-14 22:33:29 -07:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_search_timing_test.go lib/storage: skip missing tsids in the current block header by using binary search 2022-12-14 22:07:55 -08:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:31:39 -08:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
partition_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
partition_search.go lib/storage: fixes finalDedup for backfilled data (#3737) 2023-02-01 09:57:02 -08:00
partition_test.go lib/storage: remove unused code (#4050) 2023-04-02 21:23:24 -07:00
partition.go docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix 2023-07-06 22:42:02 -07:00
raw_block.go
raw_row.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
search_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
search.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
storage_test.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:23:14 -07:00
storage_timing_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
storage.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:23:14 -07:00
table_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table_search_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table_search.go lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg 2022-10-24 01:32:56 +03:00
table_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table.go lib/storage: do not create flock.lock files at partition directories, since it is created at the Storage level 2023-07-06 17:26:37 -07:00
tag_filters_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:43 +03:00
tag_filters_timing_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:43 +03:00
tag_filters.go lib/storage: optimize matching speed for non-trivial regexp filters 2022-10-01 12:07:18 +03:00
time_test.go
time.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
tsid_test.go
tsid.go