mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-26 20:30:10 +01:00
259f49089a
### Describe Your Changes
release notes for patches 1.15.6 - 9
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit ac485a8486)
485 lines
11 KiB
Markdown
485 lines
11 KiB
Markdown
---
|
|
title: Monitoring
|
|
weight: 5
|
|
menu:
|
|
docs:
|
|
parent: "vmanomaly-components"
|
|
weight: 5
|
|
identifier: "vmanomaly-monitoring"
|
|
aliases:
|
|
- ./monitoring.html
|
|
---
|
|
There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model). Parameters for each of them should be specified in the config file, `monitoring` section.
|
|
|
|
## Pull Model Config parameters
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Parameter</th>
|
|
<th>Default</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`addr`
|
|
</td>
|
|
<td>
|
|
|
|
`"0.0.0.0"`
|
|
</td>
|
|
<td>Server IP Address</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`port`
|
|
</td>
|
|
<td>
|
|
|
|
`8080`
|
|
</td>
|
|
<td>Port</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
## Push Config parameters
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Parameter</th>
|
|
<th>Default</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`url`
|
|
</td>
|
|
<td></td>
|
|
<td>
|
|
|
|
Link where to push metrics to. Example: `"http://localhost:8480/"`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`tenant_id`
|
|
</td>
|
|
<td></td>
|
|
<td>
|
|
|
|
Tenant ID for cluster version. Example: `"0:0"`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`health_path`
|
|
</td>
|
|
<td>
|
|
|
|
`"health"`
|
|
</td>
|
|
<td>
|
|
|
|
Deprecated since [v1.8.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v180). Absolute, to override `/health` path
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`user`
|
|
</td>
|
|
<td></td>
|
|
<td>BasicAuth username</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`password`
|
|
</td>
|
|
<td></td>
|
|
<td>BasicAuth password</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
`bearer_token`
|
|
</td>
|
|
<td>
|
|
`token`
|
|
</td>
|
|
<td>
|
|
Token is passed in the standard format with header: `Authorization: bearer {token}`. Available since [v1.15.9](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1159)
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
`bearer_token_file`
|
|
</td>
|
|
<td>
|
|
`path_to_file`
|
|
</td>
|
|
<td>
|
|
Path to a file, which contains token, that is passed in the standard format with header: `Authorization: bearer {token}`. Available since [v1.15.9](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1159)
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`verify_tls`
|
|
</td>
|
|
<td>
|
|
|
|
`False`
|
|
</td>
|
|
<td>Allows disabling TLS verification of the remote certificate.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`timeout`
|
|
</td>
|
|
<td>
|
|
|
|
`"5s"`
|
|
</td>
|
|
<td>Stop waiting for a response after a given number of seconds.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`extra_labels`
|
|
</td>
|
|
<td></td>
|
|
<td>Section for custom labels specified by user.</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
## Monitoring section config example
|
|
|
|
``` yaml
|
|
monitoring:
|
|
pull: # Enable /metrics endpoint.
|
|
addr: "0.0.0.0"
|
|
port: 8080
|
|
push:
|
|
url: "http://localhost:8480/"
|
|
tenant_id: "0:0" # For cluster version only
|
|
user: "USERNAME"
|
|
password: "PASSWORD"
|
|
verify_tls: False
|
|
timeout: "5s"
|
|
extra_labels:
|
|
job: "vmanomaly-push"
|
|
test: "test-1"
|
|
```
|
|
|
|
## Metrics generated by vmanomaly
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Metric</th>
|
|
<th>Type</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_start_time_seconds`
|
|
</td>
|
|
<td>Gauge</td>
|
|
<td>vmanomaly start time in UNIX time</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
### Models Behaviour Metrics
|
|
Label names [description](#labelnames)
|
|
|
|
> **Note**: There is a new label key `model_alias` introduced in multi-model support [v1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100). This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Metric</th>
|
|
<th>Type</th>
|
|
<th>Description</th>
|
|
<th>Labelnames</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_model_runs`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many times models ran (per model)</td>
|
|
<td>
|
|
|
|
`stage, query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_model_run_duration_seconds`
|
|
</td>
|
|
<td>Summary</td>
|
|
<td>How much time (in seconds) model invocations took</td>
|
|
<td>
|
|
|
|
`stage, query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_model_datapoints_accepted`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many datapoints did models accept</td>
|
|
<td>
|
|
|
|
`stage, query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_model_datapoints_produced`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many datapoints were generated by models</td>
|
|
<td>
|
|
|
|
`stage, query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_models_active`
|
|
</td>
|
|
<td>Gauge</td>
|
|
<td>How many models are currently inferring</td>
|
|
<td>
|
|
|
|
`query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_model_runs_skipped`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many times a run was skipped (per model)</td>
|
|
<td>
|
|
|
|
`stage, query_key, model_alias, scheduler_alias, preset`
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
### Writer Behaviour Metrics
|
|
Label names [description](#labelnames)
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Metric</th>
|
|
<th>Type</th>
|
|
<th>Description</th>
|
|
<th>Labelnames</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_request_duration_seconds`
|
|
</td>
|
|
<td>Summary</td>
|
|
<td>How much time (in seconds) did requests to VictoriaMetrics take</td>
|
|
<td>
|
|
|
|
`url, query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_response_count`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>Response code counts we got from VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`url, query_key, code`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_sent_bytes`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How much bytes were sent to VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`url, query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_request_serialize_seconds`
|
|
</td>
|
|
<td>Summary</td>
|
|
<td>How much time (in seconds) did serializing take</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_datapoints_sent`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many datapoints were sent to VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_writer_timeseries_sent`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many timeseries were sent to VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
### Reader Behaviour Metrics
|
|
Label names [description](#labelnames)
|
|
|
|
<table class="params">
|
|
<thead>
|
|
<tr>
|
|
<th>Metric</th>
|
|
<th>Type</th>
|
|
<th>Description</th>
|
|
<th>Labelnames</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_request_duration_seconds`
|
|
</td>
|
|
<td>Summary</td>
|
|
<td>How much time (in seconds) did queries to VictoriaMetrics take</td>
|
|
<td>
|
|
|
|
`url, query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_response_count`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>Response code counts we got from VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`url, query_key, code`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_received_bytes`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How much bytes were received in responses</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_response_parsing_seconds`
|
|
</td>
|
|
<td>Summary</td>
|
|
<td>How much time (in seconds) did parsing take for each step</td>
|
|
<td>
|
|
|
|
`step`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_timeseries_received`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many timeseries were received from VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
|
|
`vmanomaly_reader_datapoints_received`
|
|
</td>
|
|
<td>Counter</td>
|
|
<td>How many rows were received from VictoriaMetrics</td>
|
|
<td>
|
|
|
|
`query_key`
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
### Labelnames
|
|
|
|
* `stage` - stage of model - 'fit', 'infer' or 'fit_infer' for models that do it simultaneously, see [model types](https://docs.victoriametrics.com/anomaly-detection/components/models/#model-types).
|
|
* `query_key` - query alias from [`reader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/) config section.
|
|
* `model_alias` - model alias from [`models`](https://docs.victoriametrics.com/anomaly-detection/components/models/) config section. **Introduced in [v1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100).**
|
|
* `scheduler_alias` - scheduler alias from [`schedulers`](https://docs.victoriametrics.com/anomaly-detection/components/scheduler) config section. **Introduced in [v1.11.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1110).**
|
|
* `preset` - preset alias for forthcoming `preset` section compatibility. **Introduced in [v1.12.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1120).**
|
|
* `url` - writer or reader url endpoint.
|
|
* `code` - response status code or `connection_error`, `timeout`.
|
|
* `step` - json or dataframe reading step.
|