From 0b0bf94c960fb15879afba5715e49367bf21c8ea Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Tue, 18 Jan 2022 22:25:48 +0200 Subject: [PATCH] docs: update retention docs with additional details (#2060) Signed-off-by: hagen1778 --- README.md | 15 ++++++++++----- docs/README.md | 15 ++++++++++----- docs/Single-server-VictoriaMetrics.md | 15 ++++++++++----- 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 4d873636c..47d432495 100644 --- a/README.md +++ b/README.md @@ -1144,16 +1144,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. -Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. -Directories for months outside the configured retention are deleted on the first day of new month. +Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. +Data partitions outside the configured retention are deleted on the first day of new month. + +Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. +Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). + In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month. For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st. -It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower -value than before then data outside the configured period will be eventually deleted. VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days. -Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). +Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data +for multiple days and will be deleted only when fully outside of the configured retention. +It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower +value than before then data outside the configured period will be eventually deleted. ## Multiple retentions diff --git a/docs/README.md b/docs/README.md index 4d873636c..47d432495 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1144,16 +1144,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. -Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. -Directories for months outside the configured retention are deleted on the first day of new month. +Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. +Data partitions outside the configured retention are deleted on the first day of new month. + +Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. +Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). + In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month. For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st. -It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower -value than before then data outside the configured period will be eventually deleted. VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days. -Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). +Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data +for multiple days and will be deleted only when fully outside of the configured retention. +It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower +value than before then data outside the configured period will be eventually deleted. ## Multiple retentions diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index be4ba6672..e2d54d3eb 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -1148,16 +1148,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. -Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. -Directories for months outside the configured retention are deleted on the first day of new month. +Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders. +Data partitions outside the configured retention are deleted on the first day of new month. + +Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. +Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). + In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month. For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st. -It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower -value than before then data outside the configured period will be eventually deleted. VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days. -Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). +Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data +for multiple days and will be deleted only when fully outside of the configured retention. +It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower +value than before then data outside the configured period will be eventually deleted. ## Multiple retentions