From f06495c50a317946955e39536b2976d5c41cea99 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Tue, 30 Nov 2021 13:02:50 +0200 Subject: [PATCH] docs/Single-server-VictoriaMetrics.md: document that the deduplication is applied only when exporting data in JSON line format The exported data isn't de-duplicated by default due to performance reasons. It is expected that the de-duplication is applied during importing the exported data. The deduplication is applied only when exporting data via /api/v1/export if `reduce_mem_usage=1` query arg isn't passed to the request. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837 --- README.md | 9 ++++++++- docs/README.md | 9 ++++++++- docs/Single-server-VictoriaMetrics.md | 9 ++++++++- 3 files changed, 24 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 322fd9813..3292a46ba 100644 --- a/README.md +++ b/README.md @@ -789,7 +789,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val Optional `max_rows_per_line` arg may be added to the request for limiting the maximum number of rows exported per each JSON line. Optional `reduce_mem_usage=1` arg may be added to the request for reducing memory usage when exporting big number of time series. -In this case the output may contain multiple lines with distinct samples for the same time series. +In this case the output may contain multiple lines with samples for the same time series. Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: @@ -802,6 +802,9 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). +The [deduplication](#deduplication) is applied to the data exported via `/api/v1/export` by default. The deduplication +isn't applied if `reduce_mem_usage=1` query arg is passed to the request. + ### How to export CSV data @@ -826,6 +829,8 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +The [deduplication](#deduplication) isn't applied for the data exported in CSV. It is expected that the de-duplication is performed during data import. + ### How to export data in native format @@ -849,6 +854,8 @@ The exported data can be imported to VictoriaMetrics via [/api/v1/import/native] The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X can fail to be imported into VictoriaMetrics release Y. +The [deduplication](#deduplication) isn't applied for the data exported in native format. It is expected that the de-duplication is performed during data import. + ## How to import time series data diff --git a/docs/README.md b/docs/README.md index 322fd9813..3292a46ba 100644 --- a/docs/README.md +++ b/docs/README.md @@ -789,7 +789,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val Optional `max_rows_per_line` arg may be added to the request for limiting the maximum number of rows exported per each JSON line. Optional `reduce_mem_usage=1` arg may be added to the request for reducing memory usage when exporting big number of time series. -In this case the output may contain multiple lines with distinct samples for the same time series. +In this case the output may contain multiple lines with samples for the same time series. Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: @@ -802,6 +802,9 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). +The [deduplication](#deduplication) is applied to the data exported via `/api/v1/export` by default. The deduplication +isn't applied if `reduce_mem_usage=1` query arg is passed to the request. + ### How to export CSV data @@ -826,6 +829,8 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +The [deduplication](#deduplication) isn't applied for the data exported in CSV. It is expected that the de-duplication is performed during data import. + ### How to export data in native format @@ -849,6 +854,8 @@ The exported data can be imported to VictoriaMetrics via [/api/v1/import/native] The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X can fail to be imported into VictoriaMetrics release Y. +The [deduplication](#deduplication) isn't applied for the data exported in native format. It is expected that the de-duplication is performed during data import. + ## How to import time series data diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index 6dbf68690..2a253f638 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -793,7 +793,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val Optional `max_rows_per_line` arg may be added to the request for limiting the maximum number of rows exported per each JSON line. Optional `reduce_mem_usage=1` arg may be added to the request for reducing memory usage when exporting big number of time series. -In this case the output may contain multiple lines with distinct samples for the same time series. +In this case the output may contain multiple lines with samples for the same time series. Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: @@ -806,6 +806,9 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). +The [deduplication](#deduplication) is applied to the data exported via `/api/v1/export` by default. The deduplication +isn't applied if `reduce_mem_usage=1` query arg is passed to the request. + ### How to export CSV data @@ -830,6 +833,8 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +The [deduplication](#deduplication) isn't applied for the data exported in CSV. It is expected that the de-duplication is performed during data import. + ### How to export data in native format @@ -853,6 +858,8 @@ The exported data can be imported to VictoriaMetrics via [/api/v1/import/native] The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X can fail to be imported into VictoriaMetrics release Y. +The [deduplication](#deduplication) isn't applied for the data exported in native format. It is expected that the de-duplication is performed during data import. + ## How to import time series data