* Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/vmagent.html#relabeling).
## How to remove labels from metrics subset
Sometimes it may be needed to remove labels from a subset of scraped metrics, while leaving these labels in the rest of scraped metrics.
In this case the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
can be applied to `action: labeldrop` or `action: labelkeep`.
For example, the following config drops labels with names starting from `foo_` prefix from metrics matching `a{b="c"}` series selector:
- [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics)
- [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
## How to rename scraped metrics
Metric name is a regular label with special name - `__name__` (see [these docs](https://docs.victoriametrics.com/keyconcepts/#labels)).
So renaming of metric name is performed in the same way as changing label value.
Let's look at a few examples.
The following config renames `foo` metric to `bar` across all the scraped metrics, while leaving other metric names as is:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- if: 'foo'
replacement: bar
target_label: __name__
```
The following config renames metrics starting from `foo_` to metrics starting from `bar_` across all the scraped metrics. For example, `foo_count` is renamed to `bar_count`:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- source_labels: [__name__]
regex: 'foo_(.*)'
replacement: bar_$1
target_label: __name__
```
The following config replaces all the `-` chars in metric names with `_` chars across all the scraped metrics. For example, `foo-bar-baz` is renamed to `foo_bar_baz`:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- source_labels: [__name__]
action: replace_all
regex: '-'
replacement: '_'
target_label: __name__
```
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to add labels to scraped metrics
The following config sets `foo="bar"` [label](https://docs.victoriametrics.com/keyconcepts/#labels) across all the scraped metrics:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- target_label: foo
replacement: bar
```
The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- if: '{job=~"my-app-.*",env!="dev"}'
target_label: foo
replacement: bar
```
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to change label values in scraped metrics
The following config adds `foo_` prefix to all the values of `job` label across all the scraped metrics:
```yaml
scrape_configs:
- job_name: test
static_configs:
- targets: [host123]
metric_relabel_configs:
- source_labels: [job]
target_label: job
replacement: foo_$1
```
The following config adds `foo_` prefix to `job` label values only for metrics
and different values will be pushed to the storage. This is an undesired issue in most cases.
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to drop metrics during scrape
Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result
in [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality)
or [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues,
then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep`
relabeling rules at `metric_relabel_configs` section:
*`action: drop` drops all the metrics, which match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
*`action: keep` drops all the metrics, which don't match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
Note that the relabeling config is specified under `metric_relabel_configs` section instead of `relabel_configs` section:
* The `relabel_configs` is applied to the configured/discovered targets.
* The `metric_relabel_configs` is applied to metrics scraped from the configured/discovered targets.
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
## How to remove labels from a subset of targets
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/sd_configs.html),
while leaving these labels in the rest of discovered targets.
In this case the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
can be added to `action: labeldrop` or `action: labelkeep` relabeling rule.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names for targets matching `{__address__=~"pod123.+"}` selector:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "foo_$1"
- action: labeldrop
if: '{__address__=~"pod123.+"}'
regex: "foo_bar_.*"
```
See also [how to remove labels from targets](#how-to-remove-labels-from-targets).
## How to remove labels from targets
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used:
*`action: labeldrop` drops all the labels with names matching the `regex` option
*`action: labelkeep` drops all the labels with names not matching the `regex` option
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "foo_$1"
- action: labeldrop
regex: "foo_bar_.*"
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
The label name is left as is if the `regex` doesn't match it.
Important notes:
* Labels with `__` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
* Do not remove `instance` and `job` labels, since this may result in duplicate scrape targets with identical sets of labels.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to remove prefixes from target label names
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names,
while removing the `__meta_kubernetes_pod_label_` prefix. This can be done with `action: labelmap` relabeling option:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: "__meta_kubernetes_pod_label_(.+)"
replacement: "$1"
```
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
The label name is left as is if the `regex` doesn't match it.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to extract label parts
Relabeling allows extracting parts from label values and storing them into arbitrary labels.
This is performed with `regex` and `replacement` options in relabeling rules.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix:
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
The `regex` option is automatically anchored, so it must match the whole value from `source_labels`.
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
It is possible to construct a label from multiple parts of different labels. In this case just specify the needed source labels inside `source_labels` list.
The values of labels specified in `source_labels` list are joined with `;` separator by default before being matched against the `regex`.
If the `regex` doesn't match the value constructed from `source_labels`, then the relabeling rule is skipped and the remaining relabeling rules are executed.
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to modify instance and job
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) automatically add `instance` and `job` labels per each discovered target:
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-based-format)
at the resulting scrape url.
Given the scrape url construction rules above, the following config discovers pod targets
in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
and constructs per-target scrape url as `https://<pod_name>/foo/bar?baz=<container_name>`:
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to add labels to scrape targets
Additional labels can be added to scrape targets by specifying the label name in `target_label` relabeling option
and by specifying the label value in `replacement` relabeling option.
The same approach can be used for updating already existing label values at target level.
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- target_label: "foo"
replacement: "bar"
```
The labels, which are added to the target, are automatically added to all the metrics scraped from the target.
For example, if the target exposes the metric `metric{label="value"}`, then the metric is transformed into `metric{label="value",foo="bar"}`
before being sent to the storage.
If the metric exported by the target contains the same label as the target itself, then the `exported_` prefix is added to the exported label name.
For example, if the target exposes the metric `metric{foo="baz"}`, then the metric is transformed into `metric{exported_foo="baz",foo="bar"}`.
This behaviour can be changed by specifying `honor_labels: true` option at the given scrape config. In this case the exported label overrides
the target's label. In this case the `metric{foo="baz"}` stays the same. Example config with `honor_labels: true`:
```yaml
scrape_configs:
- job_name: k8s
kubernetes_sd_configs:
- role: pod
honor_labels: true
relabel_configs:
- target_label: "foo"
replacement: "bar"
```
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
## How to drop discovered targets
If a particular discovered target shouldn't be scraped, then `action: keep` or `action: drop` relabeling rules
must be used inside `relabel_configs` section.
The `action: keep` keeps only scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
and scrapes only pods with names starting with `foo` prefix:
```yaml
scrape_configs:
- job_name: foo_pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- if: '{__meta_kubernetes_pod_name=~"foo.*"}'
action: keep
```
The `action: drop` drops all the scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while keeping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
