Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 08:23:34 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

docs: follow-up after 760a530305

Browse Source 
760a530305
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e1359c904c)
This commit is contained in:
hagen1778 2023-12-01 13:27:48 +01:00
parent fbd04e3437
commit c27968e79c
No known key found for this signature in database
GPG Key ID: 3BF75F3741CA9640
1 changed files with 21 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
docs/vmagent.md
View File

@ -1225,25 +1225,35 @@ which can be downloaded for evaluation from [releases](https://github.com/Victor
The following optional suffixes are supported: s (second), m (minute), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 60s)
```
## How much disk space should you provide for the vmagent persistence queue?
## Calculating disk space for persistence queue
vmagent buffers scraped or received data at the `-remoteWrite.tmpDataPath` file system directory (aka persistent queue) until it is sent to `-remoteWrite.url`. The directory can grow large when remote storage is unavailable for extended periods of time and if the maximum directory size isn't limited with `-remoteWrite.maxDiskUsagePerURL` command-line flag. The buffered metrics are sent to remote storage as soon as the connection to the remote storage is repaired.
vmagent buffers scraped or received data in the file system directory specified via `-remoteWrite.tmpDataPath` (aka persistent queue)
until data is sent to `-remoteWrite.url`. The directory can grow large when remote storage is unavailable for extended
periods of time and if the maximum directory size isn't limited with `-remoteWrite.maxDiskUsagePerURL` command-line flag.
The buffered metrics are sent to remote storage as soon as the connection is repaired.
To estimate the disk size allocated or `-remoteWrite.maxDiskUsagePerURL` for a persistent queue take into account the following attributes:
1. the size in bytes of data stream sent by vmagent:
Run query `sum(rate(vmagent_remotewrite_bytes_sent_total[1h])) by(instance)` in [vmui](https://docs.victoriametrics.com/#vmui) or Grafana to get the amount of **bytes** sent by each vmagent instance (in k8s you might want to `sum by(pod)` instead) per second.
2. the amount of **time** a persistent queue should keep the data before starting to drop it.
If the persistent queue should be able to retain the data for at least 6h, then multiply the amount of bytes sent by vmagent per-second by 6*3600s to get the approximate queue size on disk.
To estimate the allocated disk size for persistent quee, or to estimate `-remoteWrite.maxDiskUsagePerURL` command-line flag value,
take into account the following attributes:
1. The **size in bytes** of data stream sent by vmagent:
> Run query `sum(rate(vmagent_remotewrite_bytes_sent_total[1h])) by(instance)`
in [vmui](https://docs.victoriametrics.com/#vmui) or Grafana to get the amount of bytes sent by each vmagent instance
(in k8s you might want to `sum by(pod)` instead) per second.
For example, expression [sum(rate(vmagent_remotewrite_bytes_sent_total[1h])) by(instance) * 6 * 3600 / 1Gi](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.expr=sum%28rate%28vmagent_remotewrite_bytes_sent_total%5B1h%5D%29%29+by%28instance%29+*+6+*+3600+%2F+1Gi&g0.range_input=30m&g0.end_input=2023-11-30T14%3A55%3A14&g0.tab=1&g0.relative_time=last_30_minutes) suggests that there are two vmagents: one needs 2Gi and second 21Gi of disk space available for persistent queue to retain data for 6h.
2. The amount of **time** a persistent queue should keep the data before starting to drop it.
> If the persistent queue should be able to retain the data for at least 6h, then multiply the amount of bytes
sent by vmagent per-second (see above) by 6*3600s to get the approximate queue size on disk.
For example, expression `sum(rate(vmagent_remotewrite_bytes_sent_total[1h])) by(instance) * 6 * 3600 / 1Gi`
would show how much disk space in Gi is needed for vmagents instances to retain the persisten queue for `6h`.
Additional notes:
1. For `vmagent_.*` metrics to be available for querying ensure that [monitoring](#monitoring) is configured.
1. Re-evaluate the estimation each time when:
* there is an increase in the amount of metrics to process
* there is a change in [relabeling rules](https://docs.victoriametrics.com/vmagent.html#relabeling) which could increase the amount of sent metrics
* you add a new `-remoteWrite.url`
* there is an increase in the vmagent's workload
* there is a change in [relabeling rules](https://docs.victoriametrics.com/vmagent.html#relabeling) which could increase the amount metrics to send
* there is a change in number of configured `-remoteWrite.url` addresses
1. The minimum disk size to allocate for the persistent queue is 500Mi per each `-remoteWrite.url` .
## Kafka integration
[Enterprise version](https://docs.victoriametrics.com/enterprise.html) of `vmagent` can read and write metrics from / to Kafka: