Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 00:13:30 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

docs/vmagent.md: add debugging scrape targets chapter

Browse Source
This commit is contained in:
Aliaksandr Valialkin 2024-02-08 16:29:28 +02:00
parent 70077b0da1
commit d0b55e359a
No known key found for this signature in database
GPG Key ID: 52C003EE2BCDB9EB
1 changed files with 45 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
docs/vmagent.md
View File

@ -729,7 +729,7 @@ Additionally, the `action: graphite` relabeling rules usually work much faster t
## Relabel debug
`vmagent` and [single-node VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
`vmagent` and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
provide the following tools for debugging target-level and metric-level relabeling:
- Target-level debugging (e.g. `relabel_configs` section at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs))
@ -751,6 +751,45 @@ provide the following tools for debugging target-level and metric-level relabeli
The link is unavailable if `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag.
The opened page shows step-by-step results for the actual metric relabeling rules applied to the given target labels.
See also [debugging scrape targets](#debugging-scrape-targets).
## Debugging scrape targets
`vmagent` and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
provide the following tools for debugging scrape targets:
- `http://vmagent:8429/targets` page, which contains information about all the targets, which are scraped at the moment.
This page helps answering the following questions:
- **Why some targets cannot be scraped?** The `last error` column contains the reason why the given target cannot be scraped.
You can also click the `endpoint` link in order open the target url in your browser.
You can also click the `response` link in order to open the target url on behalf of `vmagent`. This may be helpful when `vmagent`
is located in some isolated network.
- **Which labels the particular target has?** The `labels` column shows per-target labels. These labels are attached to all the metrics
scraped from the given target. You can also click on the target labels in order to see the original labels of the target
before applying the [relabeling](#relabeling). The original labels are unavailable if `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag.
- **Why the given target has the given set of labels?** Click the `target` link at `debug relabeling` column for the particular target in order to see step-by-step
execution of [target relabeling rules](#relabeling) applied to the original labels. This link is unavailable if `vmagent` runs
with `-promscrape.dropOriginalLabels` command-line flag.
- **How the given metrics relabeling rules are applied to scraped metrics?** Click the `metrics` link at `debug relabeling` column
for the particular target in order to see step-by-step execution of [metric relabeling rules](#relabeling) applied to the scraped metrics.
- **How many failed scrapes were for the particular target?** The `errors` column shows this value.
- **How many metrics the given target exposes?** The `samples` column shows the number of metrics scraped per each target during the last scrape.
- **How long does it take to scrape the given target?** The `duration` column shows last scrape duration per each target.
- **When was the last scrape for the given target?** The `last scrape` column shows the last time the given target was scraped.
- **How many times the given target was scraped?** The `scrapes` column shows this infromation.
- **What is the current state of the particular target?** The `state` column shows the current state of the particular target.
- `http://vmagent:8429/service-discovery` page, which contains infromation about all the [discovered targets](https://docs.victoriametrics.com/sd_configs/).
This page doesn't work if `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag.
This pages helps answering the following questions:
- **Why some targets are dropped during service discovery?** Click `debug` link at `debug relabeling` on the dropped target in order to see step-by-step
execution of [target relabeling rules](#relabeling) applied to the original labels of discovered target.
- **Why some targets contain unexpected labels?** Click `debug` link at `debug relabeling` on the dropped target in order to see step-by-step
execution of [target relabeling rules](#relabeling) applied to the original labels of discovered target.
- **What were the original labels before relabeling for a particular target?** The `discovered labels` column contains the original labels per each discovered target.
See also [relabel debug](#relabel-debug).
## Prometheus staleness markers
`vmagent` sends [Prometheus staleness markers](https://www.robustperception.io/staleness-and-promql) to `-remoteWrite.url` in the following cases:
@ -1144,7 +1183,11 @@ If you have suggestions for improvements or have found a bug - please open an is
regex: true
```
See also [general troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
See also:
- [debugging scrape targets](#debugging-scrape-targets)
- [relabel debug](#relabel-debug)
- [general troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html)
## Calculating disk space for persistence queue