Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 20:37:12 +01:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity
77f76371d0
VictoriaMetrics/docs/relabeling.md
Corporte Gadfly d5171c155c
docs: typo fix (#3839)
2023-02-20 10:06:19 +01:00

473 lines
23 KiB
Markdown
Raw Blame History

---
sort: 25
---
# Relabeling cookbook
VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) support
[Prometheus-compatible relabeling](https://docs.victoriametrics.com/vmagent.html#relabeling)
with [additional enhancements](https://docs.victoriametrics.com/vmagent.html#relabeling-enhancements).
The relabeling is mostly used for the following tasks:
* Dropping unneeded scrape targets during [service discovery](https://docs.victoriametrics.com/sd_configs.html#prometheus-service-discovery).
See [how to drop unneded targets with relabeling](#how-to-drop-discovered-targets).
* Adding or updating static labels at scrape targets. See [how to add labels to scrape targets](#how-to-add-labels-to-scrape-targets).
* Copying target labels from another labels. See [how to copy labels in scrape targets](#how-to-copy-labels-in-scrape-targets).
* Modifying scrape urls for discovered targets. See [how to modify scrape urls in targets](#how-to-modify-scrape-urls-in-targets).
* Modifying `instance` and `job` labels. See [how to modify instance and job](#how-to-modify-instance-and-job).
* Extracting label parts into another labels. See [how to extract label parts](#how-to-extract-label-parts).
* Removing prefixes from target label names. See [how to remove prefixes from target label names](#how-to-remove-prefixes-from-target-label-names).
* Removing some labels from discovered targets. See [how to remove labels from targets](#how-to-remove-labels-from-targets).
* Dropping some metrics during scape. See [how to drop metrics during scrape](#how-to-drop-metrics-during-scrape).
* Removing some labels from scraped metrics. See [how to remove labels from metrics](#how-to-remove-labels-from-metrics).
* Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/vmagent.html#relabeling).
## How to remove labels from metrics subset
Sometimes it may be needed to remove labels from a subset of scraped metrics, while leaving these labels in the rest of scraped metrics.
In this case the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
can be applied to `action: labeldrop` or `action: labelkeep`.
For example, the following config drops labels with names starting from `foo_` prefix from metrics matching `a{b="c"}` series selector:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- action: labeldrop
if: 'a{b="c"}'
regex: "foo_.*"
```
See also [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics).
## How to remove labels from scraped metrics
Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels
lead to [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality)
or [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues,
then it may be a good idea to drop these labels during scrapes.
This can be done with `action: labeldrop` or `action: labelkeep` relabeling rules at `metric_relabel_configs` section:
* `action: labeldrop` drops labels with names matching the given `regex` option
* `action: labelkeep` drops labels with names not matching the given `regex` option
For example, the following config drops labels with names starting with `foo_` prefix from all the metrics scraped from the `http://host123/metrics`:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- action: labeldrop
regex: "foo_.*"
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
The label name is left as is if the `regex` doesn't match it.
Important notes:
* Labels with `__` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
* Make sure that metrics exposed by the target can be uniquely identified by their names
and the remaining labels after label removal. Otherwise duplicate metrics with duplicate timestams
and different values will be pushed to the storage. This is an undesired issue in most cases.
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to drop metrics during scrape
Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result
in [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality)
or [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues,
then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep`
relabeling rules at `metric_relabel_configs` section:
* `action: drop` drops all the metrics, which match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
* `action: keep` drops all the metrics, which don't match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
For example, the following config drops all the metrics obtained from `http://host123/metrics`, which start with `foo_` prefix:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- if: '{__name__=~"foo_.*"}'
action: drop
```
Note that the relabeling config is specified under `metric_relabel_configs` section instead of `relabel_configs` section:
* The `relabel_configs` is applied to the configured/discovered targets.
* The `metric_relabel_configs` is applied to metrics scraped from the configured/discovered targets.
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to remove labels from a subset of targets
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/sd_configs.html),
while leaving these labels in the rest of discovered targets.
In this case the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
can be added to `action: labeldrop` or `action: labelkeep` relabeling rule.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names for targets matching `{__address__=~"pod123.+"}` selector:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "foo_$1"
- action: labeldrop
if: '{__address__=~"pod123.+"}'
regex: "foo_bar_.*"
```
See also [how to remove labels from targets](#how-to-remove-labels-from-targets).
## How to remove labels from targets
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used:
* `action: labeldrop` drops all the labels with names matching the `regex` option
* `action: labelkeep` drops all the labels with names not matching the `regex` option
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "foo_$1"
- action: labeldrop
regex: "foo_bar_.*"
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
The label name is left as is if the `regex` doesn't match it.
Important notes:
* Labels with `__` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
* Do not remove `instance` and `job` labels, since this may result in duplicate scrape targets with identical sets of labels.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to remove prefixes from target label names
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names,
while removing the `__meta_kubernetes_pod_label_` prefix. This can be done with `action: labelmap` relabeling option:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "$1"
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
The label name is left as is if the `regex` doesn't match it.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to extract label parts
Relabeling allows extracting parts from label values and storing them into arbitrary labels.
This is performed with `regex` and `replacement` options in relabeling rules.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name]
regex: "[^/]+/(.+)"
replacement: "abc_$1"
target_label: xyz
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is automatically anchored, so it must match the whole value from `source_labels`.
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
It is possible to construct a label from multiple parts of different labels. In this case just specify the needed source labels inside `source_labels` list.
The values of labels specified in `source_labels` list are joined with `;` separator by default before being matched against the `regex`.
The separator can be overriden via `separator` option.
If the `regex` doesn't match the value constructed from `source_labels`, then the relabeling rule is skipped and the remaining relabeling rules are executed.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to modify instance and job
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) automatically add `instance` and `job` labels per each discovered target:
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
* The `instance` label is set to the `host:port` part of `__address__` label value after target-level relabeling.
The `__address__` label value is automatically set to the most suitable value depending
on the used [service discovery type](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs).
The `__address__` label can be overriden during relabeling - see [these docs](#how-to-modify-scrape-urls-in-targets).
Both `instance` and `job` labels can be overriden during relabeling. For example, the following config discovers pod targets
in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- target_label: job
replacement: foo
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to modify scrape urls in targets
URLs for scrape targets are composed of the following parts:
* Scheme (e.g. `http` or `https`). The scheme is available during target relabeling in a special label - `__scheme__`.
By default the scheme is set to `http`. It can be overriden either by specifying the `scheme` option
at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) level
or by updating the `__scheme__` label during relabeling.
* Host and port (e.g. `host12:3456`). This information is available during target relabeling in a special label - `__address__`.
Every [supported service discovery type](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs)
sets the `__address__` label to the most suitable value. Sometimes this value needs to be modified. In this case
just update the `__address__` label during relabeling to the needed value.
The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending
on the used scheme (`http` or `https`).
The `host:port` part from the final `__address__` label is automatically set to `instance` label unless the `instance`
label is explicitly set during relabeling.
The `__address__` label can contain the full scrape url, e.g. `http://host:port/metrics/path?query_args`.
In this case the `__scheme__` and `__metrics_path__` labels are ignored.
* URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `__metrics_path__`.
By default the `__metrics_path__` is set to `/metrics`. It can be overriden either by specifying the `metrics_path`
option at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
or by updating the `__metrics_path__` label during relabeling.
* Query args (e.g. `?foo=bar&baz=xyz`). This information is available during target relabeling in special labels
with `__param_` prefix. For example, `__param_foo` would have the `bar` value, while `__param_baz` would have the `xyz` value
for `?foo=bar&baz=xyz` query string. The query args can be specified either via `params` section
at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
or by updating/setting the corresponding `__param_*` labels during relabeling.
The resulting scrape url looks like the following:
```
<__scheme__> + "://" + <__address__> + <__metrics_path__> + <"?" + query_args_from_param_labels>
```
It is expected that the target exposes metrics
in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-based-format)
at the resulting scrape url.
Given the scrape url construction rules above, the following config discovers pod targets
in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
and constructs per-target scrape url as `https://<pod_name>/foo/bar?baz=<container_name>`:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
metrics_path: /foo/bar
relabel_configs:
- target_label: __scheme__
replacement: https
- source_labels: [__meta_kubernetes_pod_name]
target_label: __address__
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: __param_baz
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to copy labels in scrape targets
Labels can be copied by specifying the source labels via `source_labels` relabeling option
and specifying the target label via `target_label` relabeling option.
For example, the following config copies `__meta_kubernetes_pod_name` label to `pod` label
for all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
```
Note that the `source_labels` option accepts a list of labels in square brackets. If multiple labels are specified
in the `source_labels` list, then the specified label values are joined into a single string with `;` delimiter by default.
The delimiter can be modified by specifying it via `separator` option.
For example, the following config sets the `pod_name:container_port` value to the `host_port` label
for all the discovered pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
separator: ":"
target_label: host_port
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to add labels to scrape targets
Additional labels can be added to scrape targets by specifying the label name in `target_label` relabeling option
and by specifying the label value in `replacement` relabeling option.
The same approach can be used for updating already existing label values at target level.
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- target_label: "foo"
replacement: "bar"
```
The labels, which are added to the target, are automatically added to all the metrics scraped from the target.
For example, if the target exposes the metric `metric{label="value"}`, then the metric is transformed into `metric{label="value",foo="bar"}`
before being sent to the storage.
If the metric exported by the target contains the same label as the target itself, then the `exported_` prefix is added to the exported label name.
For example, if the target exposes the metric `metric{foo="baz"}`, then the metric is transformed into `metric{exported_foo="baz",foo="bar"}`.
This behaviour can be changed by specifying `honor_labels: true` option at the given scrape config. In this case the exported label overrides
the target's label. In this case the `metric{foo="baz"}` stays the same. Example config with `honor_labels: true`:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
honor_labels: true
relabel_configs:
- target_label: "foo"
replacement: "bar"
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to drop discovered targets
If a particular discovered target shouldn't be scraped, then `action: keep` or `action: drop` relabeling rules
must be used inside `relabel_configs` section.
The `action: keep` keeps only scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
and scrapes only pods with names starting with `foo` prefix:
```yaml
scrape_configs:
- job_name: foo_pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- if: '{__meta_kubernetes_pod_name=~"foo.*"}'
action: keep
```
The `action: drop` drops all the scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while keeping the rest of targets. For example, the followign config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
and scrapes only pods with names starting with prefixes other than `foo`:
```yaml
scrape_configs:
- job_name: not_foo_pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- if: '{__meta_kubernetes_pod_name=~"foo.*"}'
action: drop
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## Useful tips for target relabeling
* Target relabeling can be debugged by clicking the `debug` link for the needed target on the `http://vmagent:8429/target`
or on the `http://vmagent:8429/service-discovery` pages. See [these docs](https://docs.victoriametrics.com/vmagent.html#relabel-debug).
* Every discovered target contains a set of meta-labels, which start with `__meta_` prefix.
The specific sets of labels per each supported service discovery option are listed
[here](https://docs.victoriametrics.com/sd_configs.html#prometheus-service-discovery).
* Every discovered target contains additional labels with `__` prefix other than `__meta_` labels.
See [these docs](#how-to-modify-scrape-urls-in-targets) for more details.
* All the labels, which start with `__` prefix, are automatically removed from targets after the relabeling.
So it is common practice to store temporary labels with names starting with `__` during target relabeling.
* All the target-level labels are automatically added to all the metrics scraped from targets.
* The list of discovered scrape targets with all the discovered meta-labels is available at `http://vmagent:8429/service-discovery` page for `vmagent`
and at `http://victoriametrics:8428/service-discovery` page for single-node VictoriaMetrics.
* The list of active targets with the final set of labels left after relabeling is available at `http://vmagent:8429/targets` page for `vmagent`
and at `http://victoriametrics:8428/targets` page for single-node VictoriaMetrics.
## Useful tips for metric relabeling
* Metric relabeling can be debugged at `http://vmagent:8429/metric-relabel-debug` page.
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabel-debug).
* All the labels, which start with `__` prefix, are automatically removed from metrics after the relabeling.
So it is common practice to store temporary labels with names startigh with `__` during metrics relabeling.
* All the target-level labels are automatically added to all the metrics scraped from targets,
so target-level labels are available during metrics relabeling.
Reference in New Issue View Git Blame Copy Permalink