docs: re-fresh vmctl docs

* add recommendation about network bandiwdth between vmctl, source and destination
* use more relevant time series selector in examples - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5835

Signed-off-by: hagen1778 <roman@victoriametrics.com>
This commit is contained in:
hagen1778 2024-02-29 16:58:02 +01:00
parent ca8a4cfa0d
commit 812d17f588
No known key found for this signature in database
GPG Key ID: 3BF75F3741CA9640

View File

@ -11,11 +11,7 @@ aliases:
--- ---
# vmctl # vmctl
VictoriaMetrics command-line tool VictoriaMetrics command-line tool (vmctl) provides the following list of actions:
vmctl provides various useful actions with VictoriaMetrics components.
Features:
- migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API - migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API
- migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics - migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics
- migrate data from [Cortex](#migrating-data-from-cortex) to VictoriaMetrics - migrate data from [Cortex](#migrating-data-from-cortex) to VictoriaMetrics
@ -27,8 +23,7 @@ Features:
- migrate data by [Prometheus remote read protocol](#migrating-data-by-remote-read-protocol) to VictoriaMetrics - migrate data by [Prometheus remote read protocol](#migrating-data-by-remote-read-protocol) to VictoriaMetrics
- [verify](#verifying-exported-blocks-from-victoriametrics) exported blocks from VictoriaMetrics single or cluster version. - [verify](#verifying-exported-blocks-from-victoriametrics) exported blocks from VictoriaMetrics single or cluster version.
To see the full list of supported modes To see the full list of supported actions run the following command:
run the following command:
```sh ```sh
$ ./vmctl --help $ ./vmctl --help
@ -47,8 +42,8 @@ COMMANDS:
verify-block Verifies correctness of data blocks exported via VictoriaMetrics Native format. See https://docs.victoriametrics.com/#how-to-export-data-in-native-format verify-block Verifies correctness of data blocks exported via VictoriaMetrics Native format. See https://docs.victoriametrics.com/#how-to-export-data-in-native-format
``` ```
Each mode has its own unique set of flags specific (e.g. prefixed with `influx` for influx mode) Each command has its own unique set of flags specific (e.g. prefixed with `influx-` for [influx](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x))
to the data source and common list of flags for destination (prefixed with `vm` for VictoriaMetrics): to the data source and common list of flags for destination (prefixed with `vm-` for VictoriaMetrics):
```sh ```sh
$ ./vmctl influx --help $ ./vmctl influx --help
@ -64,12 +59,11 @@ Please note, that vmctl performs initial readiness check for the given address b
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD] --vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
``` ```
When doing a migration user needs to specify flags for source (where and how to fetch data) and for When doing a migration user needs to specify flags for **source** (where and how to fetch data) and for
destination (where to migrate data). Every mode has additional details and nuances, please see **destination** (where to migrate data). Every command has additional details and nuances, please see
them below in corresponding sections. them below in corresponding sections.
For the destination flags see the full description by running the following command: For the **destination** flags see the full description by running the following command:
``` ```
$ ./vmctl influx --help | grep vm- $ ./vmctl influx --help | grep vm-
``` ```
@ -797,30 +791,29 @@ requires an Authentication header like `X-Scope-OrgID`. You can define it via th
## Migrating data from VictoriaMetrics ## Migrating data from VictoriaMetrics
The simplest way to migrate data between VM instances is to copy data. The simplest way to migrate data between VM instances is [to copy data between instances](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#data-migration).
See more details [here](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#data-migration).
vmctl uses [native binary protocol](https://docs.victoriametrics.com/#how-to-export-data-in-native-format) vmctl uses [native binary protocol](https://docs.victoriametrics.com/#how-to-export-data-in-native-format)
(available since [1.42.0 release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.42.0)) (available since [1.42.0 release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.42.0))
o migrate data between VM instances: single to single, cluster to cluster, single to cluster and vice versa. to migrate data between VM instances: single to single, cluster to cluster, single to cluster and vice versa.
See `./vmctl vm-native --help` for details and full list of flags. See `./vmctl vm-native --help` for details and full list of flags.
Migration in `vm-native` mode takes two steps: Migration in `vm-native` mode takes two steps:
1. Explore the list of the metrics to migrate via `api/v1/label/__name__/values` API; 1. Explore the list of the metrics to migrate via `api/v1/label/__name__/values` API;
1. Migrate explored metrics one-by-one. 1. Migrate explored metrics one-by-one with specified `--vm-concurrency`.
```sh ```sh
./vmctl vm-native \ ./vmctl vm-native \
--vm-native-src-addr=http://127.0.0.1:8481/select/0/prometheus \ # migrate from --vm-native-src-addr=http://127.0.0.1:8481/select/0/prometheus \ # migrate from
--vm-native-dst-addr=http://localhost:8428 \ # migrate to --vm-native-dst-addr=http://localhost:8428 \ # migrate to
--vm-native-filter-time-start='2022-11-20T00:00:00Z' \ # starting from --vm-native-filter-time-start='2022-11-20T00:00:00Z' \ # starting from
--vm-native-filter-match='{__name__=~"vm_cache_.*"}' # only metrics matching the selector --vm-native-filter-match='{__name__!~"vm_.*"}' # filter out metrics matching the selector
VictoriaMetrics Native import mode VictoriaMetrics Native import mode
2023/03/02 09:22:02 Initing import process from "http://127.0.0.1:8481/select/0/prometheus/api/v1/export/native" 2023/03/02 09:22:02 Initing import process from "http://127.0.0.1:8481/select/0/prometheus/api/v1/export/native"
to "http://localhost:8428/api/v1/import/native" with filter to "http://localhost:8428/api/v1/import/native" with filter
filter: match[]={__name__=~"vm_cache_.*"} filter: match[]={__name__!~"vm_.*"}
start: 2022-11-20T00:00:00Z start: 2022-11-20T00:00:00Z
2023/03/02 09:22:02 Exploring metrics... 2023/03/02 09:22:02 Exploring metrics...
Found 9 metrics to import. Continue? [Y/n] Found 9 metrics to import. Continue? [Y/n]
@ -841,6 +834,8 @@ _To disable explore phase and switch to the old way of data migration via single
Importing tips: Importing tips:
1. vmctl acts as a proxy between `src` and `dst`. It doesn't use much of CPU or RAM, but network connection
between `src`=>vmctl=>`dst` should be as fast as possible for improving the migration speed.
1. Migrating big volumes of data may result in reaching the safety limits on `src` side. 1. Migrating big volumes of data may result in reaching the safety limits on `src` side.
Please verify that `-search.maxExportDuration` and `-search.maxExportSeries` were set with Please verify that `-search.maxExportDuration` and `-search.maxExportSeries` were set with
proper values for `src`. If hitting the limits, follow the recommendations proper values for `src`. If hitting the limits, follow the recommendations
@ -892,7 +887,7 @@ Importing tips:
1. `vmctl` supports `--vm-native-src-headers` and `--vm-native-dst-headers` to define headers sent with each request 1. `vmctl` supports `--vm-native-src-headers` and `--vm-native-dst-headers` to define headers sent with each request
to the corresponding source address. to the corresponding source address.
1. `vmctl` supports `--vm-native-disable-http-keep-alive` to allow `vmctl` to use non-persistent HTTP connections to avoid 1. `vmctl` supports `--vm-native-disable-http-keep-alive` to allow `vmctl` to use non-persistent HTTP connections to avoid
error `use of closed network connection` when run a longer export. error `use of closed network connection` when running a heavy export requests.
### Using time-based chunking of migration ### Using time-based chunking of migration
@ -913,11 +908,11 @@ Usage example:
--vm-native-dst-addr=http://localhost:8428 \ --vm-native-dst-addr=http://localhost:8428 \
--vm-native-filter-time-start='2022-11-20T00:00:00Z' \ --vm-native-filter-time-start='2022-11-20T00:00:00Z' \
--vm-native-step-interval=month \ --vm-native-step-interval=month \
--vm-native-filter-match='{__name__=~"vm_cache_.*"}' --vm-native-filter-match='{__name__!~"vm_.*"}'
VictoriaMetrics Native import mode VictoriaMetrics Native import mode
2023/03/02 09:18:05 Initing import process from "http://127.0.0.1:8481/select/0/prometheus/api/v1/export/native" to "http://localhost:8428/api/v1/import/native" with filter 2023/03/02 09:18:05 Initing import process from "http://127.0.0.1:8481/select/0/prometheus/api/v1/export/native" to "http://localhost:8428/api/v1/import/native" with filter
filter: match[]={__name__=~"vm_cache_.*"} filter: match[]={__name__!~"vm_.*"}
start: 2022-11-20T00:00:00Z start: 2022-11-20T00:00:00Z
2023/03/02 09:18:05 Exploring metrics... 2023/03/02 09:18:05 Exploring metrics...
Found 9 metrics to import. Continue? [Y/n] Found 9 metrics to import. Continue? [Y/n]
@ -1016,14 +1011,10 @@ to number of free CPU cores.
### VictoriaMetrics importer ### VictoriaMetrics importer
The flag `--vm-concurrency` controls the number of concurrent workers that process the input from InfluxDB query results. The flag `--vm-concurrency` controls the number of concurrent workers that process the import requests to **destination**.
Please note that each import request can load up to a single vCPU core on VictoriaMetrics. So try to set it according Please note that each import request can load up to a single vCPU core on VictoriaMetrics. So try to set it according
to allocated CPU resources of your VictoriaMetrics installation. to allocated CPU resources of your VictoriaMetrics installation.
The flag `--vm-batch-size` controls max amount of samples collected before sending the import request.
For example, if `--influx-chunk-size=500` and `--vm-batch-size=2000` then importer will process not more
than 4 chunks before sending the request.
### Importer stats ### Importer stats
After successful import `vmctl` prints some statistics for details. After successful import `vmctl` prints some statistics for details.