VictoriaMetrics/app
Zakhar Bessarab af3922b1df
lib/storage: add ability to use downsampling for the given series filter (#733)
* lib/storage: add ability to use downsampling for the given series filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add information about downsampling filters

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: fix MetricsQL filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: treat missing downsampling filter as a bug

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/part_header: verify correctness of downsampling filters when opening partition

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: save only appliable rules in part metadata

Filter and save only rules which are appliable to partition based on MinTimestamp of stored data.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: update log messages for final dedup

Properly specify a reason of re-running deduplication for partition.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage: consistently use MaxTimestamp to determine deduplication/downsampling rules

Using MinTimestamp leads to applying downsampling to parts which are only partially covered by downsampling rule.
For example, partition covers range [1000-2000]. At t=2100 and rule offset 500 data with t=2100-500 => 1600 must be downsampled. The range check against MinTimestamp evaluates to true even though partition contains range which must not be downsampled - [1600:2000].

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Follow-up

- Apply the first matching downsampling period if multiple filters match the given time series.
  This allows fine-tuning the downsampling config for the specific needs.
- Take into account downsampling filters during search queries.
- Reduce the difference between community and enterprise branches. This should simplify further maintenance of these branches.
- Properly parse series filters with colons inside them.
- Document the feature at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-03-30 04:12:23 +02:00
..
victoria-logs all: add support for specifying multiple -httpListenAddr options 2024-02-09 03:15:04 +02:00
victoria-metrics app/{vminsert,vmagent}: allow using -streamAggr.dedupInterval without -streamAggr.config 2024-03-05 00:45:30 +02:00
vlinsert Add _stream fields log (#5068) 2023-11-17 15:58:52 +01:00
vlselect fix typo in kyiv city name 2024-03-23 21:56:31 +01:00
vlstorage app/vlselect: follow-up for 451d2abf50 2024-02-18 23:05:51 +02:00
vmagent app/vmagent: follow-up 166b97b8d0 2024-03-29 14:08:08 +01:00
vmalert app/vmalert: respect batch size limit for remote write on shutdown (#6039) 2024-03-29 14:27:50 +01:00
vmalert-tool all: add support for specifying multiple -httpListenAddr options 2024-02-09 03:15:04 +02:00
vmauth vmauth: support other auth header names besides Authorization (#6009) 2024-03-26 13:21:07 +01:00
vmbackup lib/snapshot: move Time, Validate and NewName into lib/snapshot/snapshotutil package 2024-02-09 04:18:45 +02:00
vmbackupmanager docs: convert png images to webp in all the docs except of docs/operator/* 2023-11-22 19:21:00 +02:00
vmctl app/vmctl: break explore phase in vm-native mode by time intervals 2024-03-18 12:18:32 +01:00
vmgateway docs: convert png images to webp in all the docs except of docs/operator/* 2023-11-22 19:21:00 +02:00
vminsert app/{vmagent,vminsert}: fixed firehose response (#6016) 2024-03-26 13:20:41 +01:00
vmrestore all: add support for specifying multiple -httpListenAddr options 2024-02-09 03:15:04 +02:00
vmselect lib/storage: add ability to use downsampling for the given series filter (#733) 2024-03-30 04:12:23 +02:00
vmstorage lib/storage: adds metrics for downsampling (#382) 2024-03-30 01:11:49 +02:00
vmui fix typo in kyiv city name 2024-03-23 21:56:31 +01:00