Aliaksandr Valialkin
233301a549
lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention
2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search
...
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
57ea7a3ee8
lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback
...
This improves code readability a bit.
2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin
63419d8e7c
lib/storage: small refactoring: move retentionDeadline to blockStreamMerger
...
This allows defining per-block retention in the future by updating the getRetentionDeadline function
2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin
31071347ca
lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop
2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin
d36fdbe537
lib/storage: deduplicate samples more thoroughly
...
Previously some duplicate samples may be left on disk for time series with high churn rate.
This may result in higher disk space usage.
2021-12-15 16:00:30 +02:00
Aliaksandr Valialkin
940a547116
lib/storage: do not update b.nextIdx if no samples are removed because of retention
2021-03-29 12:13:38 +03:00
Aliaksandr Valialkin
ea328b7391
lib/storage: skip deduplication when creating inmemory data blocks
...
The deduplication will be performed later during merging such blocks.
2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin
72011bcc45
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin
f2bff64933
lib/storage: remove data race when updating rowsDeleted
2020-11-05 01:19:30 +02:00
Aliaksandr Valialkin
901514be88
lib/storage: drop more samples outside the given retention during background merge
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-10-31 20:44:47 +02:00
Aliaksandr Valialkin
0db7c2b500
app/vmstorage: support for -retentionPeriod smaller than one month
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-10-20 14:42:46 +03:00
Aliaksandr Valialkin
3abbb38254
lib/{mergeset,storage}: compare errors with errors.Is()
2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin
29bbab0ec9
lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected
...
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.
Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
b8303afcd8
lib/storage: improve prioritizing of data ingestion over querying
...
Prioritize also small merges over big merges.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
23fa44e56e
lib/storage: reorder mergeBlockStreams() args in order to make them more consistent
2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
de0e4eee2c
lib/storage: create and use lib/uint64set instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
fb9358635d
lib/storage: mention source parts on merge error
...
This should improve determining broken source part.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
