update configuration recommendations for Prometheus remote_write (#277)

This commit is contained in:
Roman Khavronenko 2019-12-23 18:33:10 +00:00 committed by Aliaksandr Valialkin
parent 6a1499efa3
commit 85463a7199

View File

@ -136,14 +136,13 @@ It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics.
### Prometheus setup ### Prometheus setup
Add the following lines to Prometheus config file (it is usually located at `/etc/prometheus/prometheus.yml`): Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
in order to send data to VictoriaMetrics. Add the following lines
to Prometheus config file (it is usually located at `/etc/prometheus/prometheus.yml`):
```yml ```yml
remote_write: remote_write:
- url: http://<victoriametrics-addr>:8428/api/v1/write - url: http://<victoriametrics-addr>:8428/api/v1/write
queue_config:
max_samples_per_send: 10000
max_shards: 30
``` ```
Substitute `<victoriametrics-addr>` with the hostname or IP address of VictoriaMetrics. Substitute `<victoriametrics-addr>` with the hostname or IP address of VictoriaMetrics.
@ -170,6 +169,22 @@ This instructs Prometheus to add `datacenter=dc-123` label to each time series s
The label name may be arbitrary - `datacenter` is just an example. The label value must be unique The label name may be arbitrary - `datacenter` is just an example. The label value must be unique
across Prometheus instances, so those time series may be filtered and grouped by this label. across Prometheus instances, so those time series may be filtered and grouped by this label.
For highly loaded Prometheus instances (400k+ samples per second)
the following tuning may be applied:
```
remote_write:
- url: http://<victoriametrics-addr>:8428/api/v1/write
queue_config:
max_samples_per_send: 10000
capacity: 20000
max_shards: 30
```
Using remote write increases memory usage for Prometheus up to ~25%
and depends on the shape of data. If you are experiencing issues with
too high memory consumption try to lower `max_samples_per_send`
and `capacity` params (keep in mind that these two params are tightly connected).
Read more about tuning remote write for Prometheus [here](https://prometheus.io/docs/practices/remote_write).
It is recommended upgrading Prometheus to [v2.12.0](https://github.com/prometheus/prometheus/releases) or newer, It is recommended upgrading Prometheus to [v2.12.0](https://github.com/prometheus/prometheus/releases) or newer,
since the previous versions may have issues with `remote_write`. since the previous versions may have issues with `remote_write`.