mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-22 08:26:19 +01:00
73 lines
2.6 KiB
Markdown
73 lines
2.6 KiB
Markdown
---
|
|
sort: 19
|
|
---
|
|
|
|
## VictoriaMetrics Cluster Per Tenant Statistic
|
|
|
|
<img alt="cluster-per-tenant-stat" src="per-tenant-stats.jpg">
|
|
|
|
The enterprise version of VictoriaMetrics cluster exposes the usage statistics for each tenant.
|
|
|
|
When the next statistic is exposed:
|
|
|
|
- `vminsert`
|
|
|
|
* `vm_tenant_inserted_rows_total` - the ingestion rate by tenant
|
|
- `vmselect`
|
|
|
|
* `vm_tenant_select_requests_duration_ms_total` - query latency by tenant. It can be useful to identify the tenant with the heaviest queries
|
|
* `vm_tenant_select_requests_total` - total requests. You can calculate request rate (qps) with this metric
|
|
|
|
- `vmstorage`
|
|
* `vm_tenant_active_timeseries` - the number of active timeseries
|
|
* `vm_tenant_used_tenant_bytes` - the disk space consumed by the metrics for a particular tenant
|
|
* `vm_tenant_timeseries_created_total` - the total number for timeseries by tenant
|
|
|
|
|
|
The information should be scraped by the agent (`vmagent`, `victoriametrics`, prometheus, etc) and stored in the TSDB. This can be the same cluster but a different tenant however, we encourage the use of one more instance of TSDB (more lightweight, eg. VM single) for the monitoring of monitoring.
|
|
|
|
the config example for statistic scraping
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: cluster
|
|
scrape_interval: 10s
|
|
static_configs:
|
|
- targets: ['vmselect:8481','vmstorage:8482','vminsert:8480']
|
|
```
|
|
|
|
### Visualization
|
|
|
|
Visualisation of statistics can be done in Grafana using this dashboard [link](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster/dashboards/clusterbytenant.json)
|
|
|
|
|
|
### Integration with vmgateway
|
|
|
|
Per Tenant Statistics are the source data for the `vmgateway` rate limiter. More information can be found [here](https://docs.victoriametrics.com/vmgateway.html)
|
|
|
|
### Integration with vmalert
|
|
|
|
You can generate alerts based on each tenants' resource usage and notify the system/users that they are reaching the limits.
|
|
|
|
Here is an example of an alert for high churn rate by the tenant
|
|
|
|
```yaml
|
|
|
|
- alert: TooHighChurnRate
|
|
expr: |
|
|
(
|
|
sum(rate(vm_tenant_timeseries_created_total[5m])) by(accountID,projectID)
|
|
/
|
|
sum(rate(vm_tenant_inserted_rows_total[5m])) by(accountID,projectID)
|
|
) > 0.1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Churn rate is more than 10% for the last 15m"
|
|
description: "VM constantly creates new time series in the tenant: {{ $labels.accountID }}:{{ $labels.projectID }}.\n
|
|
This effect is known as Churn Rate.\n
|
|
High Churn Rate is tightly connected with database performance and may
|
|
result in unexpected OOM's or slow queries."
|
|
```
|