diff --git a/README.md b/README.md index 30a1e2d44..10f4eb0a9 100644 --- a/README.md +++ b/README.md @@ -765,33 +765,10 @@ when new data is ingested into it. VictoriaMetrics provides the following handlers for exporting data: -* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. - See [these docs](#how-to-export-data-in-native-format) for details. * `/api/v1/export` for exporing data in JSON line format. See [these docs](#how-to-export-data-in-json-line-format) for details. * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details. - - -### How to export data in native format - -Send a request to `http://:8428/api/v1/export/native?match[]=`, -where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) -for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. - -On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: - -```bash -# count unique timeseries in database -wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' - -# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command -``` - -Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either -unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. - -The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). -The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X -can fail to be imported into VictoriaMetrics release Y. +* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. + See [these docs](#how-to-export-data-in-native-format) for details. ### How to export data in JSON line format @@ -850,6 +827,29 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +### How to export data in native format + +Send a request to `http://:8428/api/v1/export/native?match[]=`, +where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) +for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. + +On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: + +```bash +# count unique timeseries in database +wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' + +# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command +``` + +Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either +unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. + +The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). +The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X +can fail to be imported into VictoriaMetrics release Y. + + ## How to import time series data Time series data can be imported via any supported ingestion protocol: @@ -868,36 +868,6 @@ Time series data can be imported via any supported ingestion protocol: * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details. -### How to import data in native format - -The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. - -If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. - -```bash -# Export the data from : -curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin - -# Import the data to : -curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin -``` - -Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/native` for importing gzipped data: - -```bash -# Export gzipped data from : -curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin.gz - -# Import gzipped data to : -curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin.gz -``` - -Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. -For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. - -Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. - - ### How to import data in JSON line format Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format): @@ -928,6 +898,26 @@ Note that it could be required to flush response cache after importing historica VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines. +### How to import data in native format + +The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. + +If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. + +```bash +# Export the data from : +curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin + +# Import the data to : +curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin +``` + +Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. +For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. + +Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. + + ### How to import CSV data Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg. @@ -1003,6 +993,13 @@ It should return something like the following: {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` +Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: + +```bash +# Import gzipped data to : +curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz +``` + Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics. diff --git a/docs/README.md b/docs/README.md index 30a1e2d44..10f4eb0a9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -765,33 +765,10 @@ when new data is ingested into it. VictoriaMetrics provides the following handlers for exporting data: -* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. - See [these docs](#how-to-export-data-in-native-format) for details. * `/api/v1/export` for exporing data in JSON line format. See [these docs](#how-to-export-data-in-json-line-format) for details. * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details. - - -### How to export data in native format - -Send a request to `http://:8428/api/v1/export/native?match[]=`, -where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) -for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. - -On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: - -```bash -# count unique timeseries in database -wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' - -# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command -``` - -Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either -unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. - -The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). -The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X -can fail to be imported into VictoriaMetrics release Y. +* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. + See [these docs](#how-to-export-data-in-native-format) for details. ### How to export data in JSON line format @@ -850,6 +827,29 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +### How to export data in native format + +Send a request to `http://:8428/api/v1/export/native?match[]=`, +where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) +for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. + +On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: + +```bash +# count unique timeseries in database +wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' + +# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command +``` + +Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either +unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. + +The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). +The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X +can fail to be imported into VictoriaMetrics release Y. + + ## How to import time series data Time series data can be imported via any supported ingestion protocol: @@ -868,36 +868,6 @@ Time series data can be imported via any supported ingestion protocol: * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details. -### How to import data in native format - -The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. - -If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. - -```bash -# Export the data from : -curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin - -# Import the data to : -curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin -``` - -Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/native` for importing gzipped data: - -```bash -# Export gzipped data from : -curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin.gz - -# Import gzipped data to : -curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin.gz -``` - -Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. -For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. - -Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. - - ### How to import data in JSON line format Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format): @@ -928,6 +898,26 @@ Note that it could be required to flush response cache after importing historica VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines. +### How to import data in native format + +The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. + +If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. + +```bash +# Export the data from : +curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin + +# Import the data to : +curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin +``` + +Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. +For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. + +Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. + + ### How to import CSV data Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg. @@ -1003,6 +993,13 @@ It should return something like the following: {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` +Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: + +```bash +# Import gzipped data to : +curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz +``` + Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics. diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index 2270b42c5..0aaeb162a 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -769,33 +769,10 @@ when new data is ingested into it. VictoriaMetrics provides the following handlers for exporting data: -* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. - See [these docs](#how-to-export-data-in-native-format) for details. * `/api/v1/export` for exporing data in JSON line format. See [these docs](#how-to-export-data-in-json-line-format) for details. * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details. - - -### How to export data in native format - -Send a request to `http://:8428/api/v1/export/native?match[]=`, -where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) -for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. - -On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: - -```bash -# count unique timeseries in database -wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' - -# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command -``` - -Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either -unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. - -The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). -The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X -can fail to be imported into VictoriaMetrics release Y. +* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export. + See [these docs](#how-to-export-data-in-native-format) for details. ### How to export data in JSON line format @@ -854,6 +831,29 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). +### How to export data in native format + +Send a request to `http://:8428/api/v1/export/native?match[]=`, +where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) +for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. + +On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: + +```bash +# count unique timeseries in database +wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' + +# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command +``` + +Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either +unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values. + +The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). +The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X +can fail to be imported into VictoriaMetrics release Y. + + ## How to import time series data Time series data can be imported via any supported ingestion protocol: @@ -872,36 +872,6 @@ Time series data can be imported via any supported ingestion protocol: * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details. -### How to import data in native format - -The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. - -If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. - -```bash -# Export the data from : -curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin - -# Import the data to : -curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin -``` - -Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/native` for importing gzipped data: - -```bash -# Export gzipped data from : -curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin.gz - -# Import gzipped data to : -curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin.gz -``` - -Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. -For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. - -Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. - - ### How to import data in JSON line format Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format): @@ -932,6 +902,26 @@ Note that it could be required to flush response cache after importing historica VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines. +### How to import data in native format + +The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file. + +If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in. + +```bash +# Export the data from : +curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin + +# Import the data to : +curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin +``` + +Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args. +For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series. + +Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. + + ### How to import CSV data Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg. @@ -1007,6 +997,13 @@ It should return something like the following: {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` +Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: + +```bash +# Import gzipped data to : +curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz +``` + Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.