Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-27 02:46:47 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: update remote-read api docs

Browse Source 
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This commit is contained in:
hagen1778 2023-08-25 15:21:19 +02:00
parent 0b0e0bb50e
commit f4577005be
No known key found for this signature in database
GPG Key ID: 3BF75F3741CA9640
1 changed files with 30 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
app/vmctl/README.md
View File

@ -310,12 +310,17 @@ You may find useful a 3rd party solution for this - <https://github.com/jonppe/i
## Migrating data from Promscale
[Promscale](https://github.com/timescale/promscale) supports [Prometheus Remote Read API](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/).
Hence, `vmctl` can be used in [remote-read](https://docs.victoriametrics.com/vmctl.html#migrating-data-by-remote-read-protocol)
mode to migrate historical data from Promscale to VictoriaMetrics.
To migrate historical data from Promscale to VictoriaMetrics we recommend using `vmctl`
in [remote-read](https://docs.victoriametrics.com/vmctl.html#migrating-data-by-remote-read-protocol) mode.
See the example of migration command below:
```console
./vmctl remote-read --remote-read-src-addr=http://127.0.0.1:9201/read --remote-read-step-interval=day --remote-read-use-stream=false --vm-addr=http://127.0.0.1:8428 --remote-read-filter-time-start=2023-08-21T00:00:00Z --vm-concurrency=6 --remote-read-disable-path-append=true
./vmctl remote-read --remote-read-src-addr=http://<promscale>:9201/read \
--remote-read-step-interval=day \
--remote-read-use-stream=false \ # promscale doesn't support streaming
--vm-addr=http://<victoriametrics>:8428 \
--remote-read-filter-time-start=2023-08-21T00:00:00Z \
--remote-read-disable-path-append=true # promscale has custom remote read API HTTP path
Selected time range "2023-08-21 00:00:00 +0000 UTC" - "2023-08-21 14:11:41.561979 +0000 UTC" will be split into 1 ranges according to "day" step. Continue? [Y/n] y
VM worker 0:↙ 82831 samples/s
VM worker 1:↙ 54378 samples/s
@ -337,10 +342,10 @@ Processing ranges: 1 / 1 [██████████████████
2023/08/21 16:11:55 Total time: 14.063458792s
```
Here we specify the full path to Promscale's Remote Read API via `--remote-read-src-addr` and we disable auto-path
Here we specify the full path to Promscale's Remote Read API via `--remote-read-src-addr`, and disable auto-path
appending via `--remote-read-disable-path-append` cmd-line flags. This is necessary, as Promscale has a different to
Prometheus API path. Promscale doesn't support stream mode, so it may consume additional memory during
the migration.
Prometheus API path. Promscale doesn't support stream mode for Remote Read API,
so we disable it via `--remote-read-use-stream=false`.
## Migrating data from Prometheus
@ -474,17 +479,12 @@ Found 2 blocks to import. Continue? [Y/n] y
## Migrating data by remote read protocol
`vmctl` supports the `remote-read` mode for migrating data from databases which support
`vmctl` provides the `remote-read` mode for migrating data from remote databases supporting
[Prometheus remote read API](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/).
Remote read API implemented two modes:
1. `SAMPLES` - returns a message that includes a list of raw samples (one protocol buffer message);
2. `STREAMED_XOR_CHUNKS` - sending a set of small protocol buffer messages;
`STREAMED_XOR_CHUNKS` mode uses constant memory per request no matter how many samples was requested, when `SAMPLES` mode
buffered the whole response of the remote read a raw, uncompressed format in order to marshsal it in a potentially
huge protobuf message before sending it to the client. So potentially `SAMPLES` mode may lead to allocate a lot of memory
by both client and server each.
For more information, please check this [blog](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/).
Remote read API has two implementations of remote read API: default (`SAMPLES`) and
[streamed](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/) (`STREAMED_XOR_CHUNKS`).
Streamed version is more efficient but has lower adoption (e.g. [Promscale](#migrating-data-from-promscale)
doesn't support it).
See `./vmctl remote-read --help` for details and full list of flags.
@ -502,10 +502,10 @@ and single-node VictoriaMetrics(`http://localhost:8428`):
```
./vmctl remote-read \
--remote-read-src-addr=http://127.0.0.1:9091 \
--remote-read-src-addr=http://<prometheus>:9091 \
--remote-read-filter-time-start=2021-10-18T00:00:00Z \
--remote-read-step-interval=hour \
--vm-addr=http://127.0.0.1:8428 \
--vm-addr=http://<victoria-metrics>:8428 \
--vm-concurrency=6
Split defined times into 8798 ranges to import. Continue? [Y/n]
@ -704,26 +704,24 @@ requires an Authentication header like `X-Scope-OrgID`. You can define it via th
## Migrating data from Mimir
Mimir has similar implementation as Cortex and also support of the Prometheus remote read protocol. That means
`vmctl` in mode `remote-read` may also be used for Mimir historical data migration.
These instructions may vary based on the details of your Mimir configuration.
Mimir has similar implementation as Cortex and supports Prometheus remote read API. That means historical data
from Mimir can be migrated via `vmctl` in mode `remote-read` mode.
The instructions for data migration via vmctl vary based on the details of your Mimir configuration.
Please read carefully and verify as you go.
### Remote read protocol
If you want to migrate data, you should check your Mimir configuration in the section
```yaml
api:
prometheus_http_prefix:
```
By default, Mimir uses the `prometheus` path prefix so specifying the source
should be as simple as `--remote-read-src-addr=http://<mimir>:9009/prometheus`.
But if prefix was overriden via `prometheus_http_prefix`, then source address should be updated
to `--remote-read-src-addr=http://<mimir>:9009/{prometheus_http_prefix}`.
If you defined some prometheus prefix, you should use it when you define flag `--remote-read-src-addr=http://127.0.0.1:9009/{prometheus_http_prefix}`.
By default, Mimir uses the `prometheus` path prefix, so you should define the flag `--remote-read-src-addr=http://127.0.0.1:9009/prometheus`.
Mimir supports [streamed remote read API](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/),
so it is recommended setting `--remote-read-use-stream=true` flag for better performance and resource usage.
Mimir supports both remote read mode, so you can use stream mode by setting `--remote-read-use-stream=true` flag.
When you run Mimir, it exposes a port to serve HTTP on `8080 by default`.
Next example of the local installation was in multi-tenant mode (3 instances of mimir) with nginx as load balancer.
Next example of the local installation was in multi-tenant mode (3 instances of Mimir) with nginx as load balancer.
Load balancer expose single port `:9090`.
As you can see in the example we call `:9009` instead of `:8080` because of proxy.
@ -733,13 +731,12 @@ and single-node VictoriaMetrics(`http://localhost:8428`):
```
./vmctl remote-read
--remote-read-src-addr=http://127.0.0.1:9009/prometheus \
--remote-read-src-addr=http://<mimir>:9009/prometheus \
--remote-read-filter-time-start=2021-10-18T00:00:00Z \
--remote-read-step-interval=hour \
--remote-read-headers=X-Scope-OrgID:demo \
--remote-read-use-stream=true \
--vm-addr=http://127.0.0.1:8428 \
--vm-concurrency=6
--vm-addr=http://<victoria-metrics>:8428 \
```
And when the process finishes, you will see the following: