mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-23 12:31:07 +01:00
docs/relabeling.md: add a cookbook for common relabeling tasks
This commit is contained in:
Aliaksandr Valialkin
parent
31f922944e
commit
43c6f81b90
@ -1139,7 +1139,11 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
|
||||
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
|
||||
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
|
||||
The `-relabelConfig` also can point to http or https url. For example, `-relabelConfig=https://config-server/relabel_config.yml`.
|
||||
See [this article with relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The following docs can be useful in understanding the relabeling:
|
||||
|
||||
* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling.html).
|
||||
* [Relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The `-relabelConfig` files can contain special placeholders in the form `%{ENV_VAR}`, which are replaced by the corresponding environment variable values.
|
||||
|
||||
|
||||
@ -314,7 +314,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
|
||||
|
||||
## Relabeling
|
||||
|
||||
VictoriaMetrics components (including `vmagent`) support [Prometheus-compatible relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) with [additional enhancements](#relabeling-enhancements) at various stages of data processing. The relabeling can be defined in the following places processed by `vmagent`:
|
||||
VictoriaMetrics components support [Prometheus-compatible relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) with [additional enhancements](#relabeling-enhancements) at various stages of data processing. The relabeling can be defined in the following places processed by `vmagent`:
|
||||
|
||||
* At the `scrape_config -> relabel_configs` section in `-promscrape.config` file. This relabeling is used for modifying labels in discovered targets and for dropping unneded targets. This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
|
||||
* At the `scrape_config -> metric_relabel_configs` section in `-promscrape.config` file. This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics. This relabeling can be debugged by passing `metric_relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs metrics before and after the relabeling and then drops the logged metrics.
|
||||
@ -325,6 +325,7 @@ All the files with relabeling configs can contain special placeholders in the fo
|
||||
|
||||
The following articles contain useful information about Prometheus relabeling:
|
||||
|
||||
* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling.html)
|
||||
* [How to use Relabeling in Prometheus and VictoriaMetrics](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2)
|
||||
* [Life of a label](https://www.robustperception.io/life-of-a-label)
|
||||
* [Discarding targets and timeseries with relabeling](https://www.robustperception.io/relabelling-can-discard-targets-timeseries-and-alerts)
|
||||
|
||||
@ -1139,7 +1139,11 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
|
||||
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
|
||||
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
|
||||
The `-relabelConfig` also can point to http or https url. For example, `-relabelConfig=https://config-server/relabel_config.yml`.
|
||||
See [this article with relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The following docs can be useful in understanding the relabeling:
|
||||
|
||||
* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling.html).
|
||||
* [Relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The `-relabelConfig` files can contain special placeholders in the form `%{ENV_VAR}`, which are replaced by the corresponding environment variable values.
|
||||
|
||||
|
||||
@ -1143,7 +1143,11 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
|
||||
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
|
||||
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
|
||||
The `-relabelConfig` also can point to http or https url. For example, `-relabelConfig=https://config-server/relabel_config.yml`.
|
||||
See [this article with relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The following docs can be useful in understanding the relabeling:
|
||||
|
||||
* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling.html).
|
||||
* [Relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2).
|
||||
|
||||
The `-relabelConfig` files can contain special placeholders in the form `%{ENV_VAR}`, which are replaced by the corresponding environment variable values.
|
||||
|
||||
|
||||
461
docs/relabeling.md
Normal file
461
docs/relabeling.md
Normal file
@ -0,0 +1,461 @@
|
||||
# Relabeling
|
||||
|
||||
VicotriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) support
|
||||
[Prometheus-compatible relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config)
|
||||
with additional enhancements.
|
||||
|
||||
The relabeling is mostly used for the following tasks:
|
||||
|
||||
* Dropping unneeded scrape targets during [service discovery](https://docs.victoriametrics.com/sd_configs.html#prometheus-service-discovery).
|
||||
See [how to drop unneded targets with relabeling](#how-to-drop-discovered-targets).
|
||||
* Adding or updating static labels at scrape targets. See [how to add labels to scrape targets](#how-to-add-labels-to-scrape-targets).
|
||||
* Copying target labels from another labels. See [how to copy labels in scrape targets](#how-to-copy-labels-in-scrape-targets).
|
||||
* Modifying scrape urls for discovered targets. See [how to modify scrape urls in targets](#how-to-modify-scrape-urls-in-targets).
|
||||
* Modifying `instance` and `job` labels. See [how to modify instance and job](#how-to-modify-instance-and-job).
|
||||
* Extracting label parts into another labels. See [how to extract label parts](#how-to-extract-label-parts).
|
||||
* Removing prefixes from target label names. See [how to remove prefixes from target label names](#how-to-remove-prefixes-from-target-label-names).
|
||||
* Removing some labels from discovered targets. See [how to remove labels from targets](#how-to-remove-labels-from-targets).
|
||||
* Dropping some metrics during scape. See [how to drop metrics during scrape](#how-to-drop-metrics-during-scrape).
|
||||
* Removing some labels from scraped metrics. See [how to remove labels from metrics](#how-to-remove-labels-from-metrics).
|
||||
* Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
|
||||
See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
|
||||
|
||||
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/vmagent.html#relabeling).
|
||||
|
||||
|
||||
## How to remove labels from metrics subset
|
||||
|
||||
Sometimes it may be needed to remove labels from a subset of scraped metrics, while leaving these labels in the rest of scraped metrics.
|
||||
In this case the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
can be applied to `action: labeldrop` or `action: labelkeep`.
|
||||
|
||||
For example, the following config drops labels with names starting from `foo_` prefix from metrics matching `a{b="c"}` series selector:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: test
|
||||
static_configs:
|
||||
- targets: [host123]
|
||||
metric_relabel_configs:
|
||||
- action: labeldrop
|
||||
if: 'a{b="c"}'
|
||||
regex: "foo_.*"
|
||||
```
|
||||
|
||||
See also [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics).
|
||||
|
||||
|
||||
## How to remove labels from scraped metrics
|
||||
|
||||
Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels
|
||||
lead to [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues,
|
||||
then it may be a good idea to drop these labels during scrapes.
|
||||
This can be done with `action: labeldrop` or `action: labelkeep` relabeling rules at `metric_relabel_configs` section:
|
||||
|
||||
* `action: labeldrop` drops labels with names matching the given `regex` option
|
||||
* `action: labelkeep` drops labels with names not matching the given `regex` option
|
||||
|
||||
For example, the following config drops labels with names starting with `foo_` prefix from all the metrics scraped from the `http://host123/metrics`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: test
|
||||
static_configs:
|
||||
- targets: [host123]
|
||||
metric_relabel_configs:
|
||||
- action: labeldrop
|
||||
regex: "foo_.*"
|
||||
```
|
||||
|
||||
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
|
||||
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
|
||||
The label name is left as is if the `regex` doesn't match it.
|
||||
|
||||
Important notes:
|
||||
|
||||
* Labels with `__` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
|
||||
* Make sure that metrics exposed by the target can be uniquely identified by their names
|
||||
and the remaining labels after label removal. Otherwise duplicate metrics with duplicate timestams
|
||||
and different values will be pushed to the storage. This is an undesired issue in most cases.
|
||||
|
||||
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
|
||||
|
||||
|
||||
## How to drop metrics during scrape
|
||||
|
||||
Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result
|
||||
in [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) issues,
|
||||
then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep`
|
||||
relabeling rules at `metric_relabel_configs` section:
|
||||
|
||||
* `action: drop` drops all the metrics, which match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
* `action: keep` drops all the metrics, which don't match the `if` [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
|
||||
For example, the following config drops all the metrics obtained from `http://host123/metrics`, which start from `foo_` prefix:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: test
|
||||
static_configs:
|
||||
- targets: [host123]
|
||||
metric_relabel_configs:
|
||||
- if: '{__name__=~"foo_.*"}'
|
||||
action: drop
|
||||
```
|
||||
|
||||
Note that the relabeling config is specified under `metric_relabel_configs` section instead of `relabel_configs` section:
|
||||
|
||||
* The `relabel_configs` is applied to the configured/discovered targets.
|
||||
* The `metric_relabel_configs` is applied to metrics scraped from the configured/discovered targets.
|
||||
|
||||
See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
|
||||
|
||||
|
||||
## How to remove labels from a subset of targets
|
||||
|
||||
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/sd_configs.html),
|
||||
while leaving these labels in the rest of discovered targets.
|
||||
In this case the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
can be added to `action: labeldrop` or `action: labelkeep` relabeling rule.
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
|
||||
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
|
||||
with `foo_bar_` prefix in their names for targets matching `{__address__=~"pod123.+"}` selector:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: "__meta_kubernetes_pod_label_(.+)"
|
||||
replacement: "foo_$1"
|
||||
- action: labeldrop
|
||||
if: '{__address__=~"pod123.+"}'
|
||||
regex: "foo_bar_.*"
|
||||
```
|
||||
|
||||
See also [how to remove labels from targets](#how-to-remove-labels-from-targets).
|
||||
|
||||
|
||||
## How to remove labels from targets
|
||||
|
||||
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
|
||||
In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used:
|
||||
|
||||
* `action: labeldrop` drops all the labels with names matching the `regex` option
|
||||
* `action: labelkeep` drops all the labels with names not matching the `regex` option
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
|
||||
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
|
||||
with `foo_bar_` prefix in their names:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: "__meta_kubernetes_pod_label_(.+)"
|
||||
replacement: "foo_$1"
|
||||
- action: labeldrop
|
||||
regex: "foo_bar_.*"
|
||||
```
|
||||
|
||||
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
|
||||
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
|
||||
The label name is left as is if the `regex` doesn't match it.
|
||||
|
||||
Important notes:
|
||||
|
||||
* Labels with `__` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
|
||||
* Do not remove `instance` and `job` labels, since this may result in duplicate scrape targets with identical sets of labels.
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to remove prefixes from target label names
|
||||
|
||||
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/sd_configs.html).
|
||||
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
|
||||
labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names,
|
||||
while removing the `__meta_kubernetes_pod_label_` prefix. This can be done with `action: labelmap` relabeling option:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: "__meta_kubernetes_pod_label_(.+)"
|
||||
replacement: "$1"
|
||||
```
|
||||
|
||||
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
|
||||
The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name.
|
||||
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
|
||||
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
|
||||
|
||||
The label name is left as is if the `regex` doesn't match it.
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to extract label parts
|
||||
|
||||
Relabeling allows extracting parts from label values and storing them into arbitrary labels.
|
||||
This is performed with `regex` and `replacement` options in relabeling rules.
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs),
|
||||
extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
regex: "[^/]+/(.+)"
|
||||
replacement: "abc_$1"
|
||||
target_label: xyz
|
||||
```
|
||||
|
||||
The `regex` option can contain arbitrary regular expression in [RE2 format](https://github.com/google/re2/wiki/Syntax).
|
||||
The `regex` option is automatically anchored, so it must match the whole value from `source_labels`.
|
||||
It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option
|
||||
with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference.
|
||||
|
||||
It is possible to construct a label from multiple parts of different labels. In this case just specify the needed source labels inside `source_labels` list.
|
||||
The values of labels specified in `source_labels` list are joined with `;` separator by default before being matched against the `regex`.
|
||||
The separator can be overriden via `separator` option.
|
||||
|
||||
If the `regex` doesn't match the value constructed from `source_labels`, then the relabeling rule is skipped and the remaining relabeling rules are executed.
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to modify instance and job
|
||||
|
||||
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent.html) automatically add `instance` and `job` labels per each discovered target:
|
||||
|
||||
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
|
||||
* The `instance` label is set to the final `__address__` label value after target-level relabeling.
|
||||
The `__address__` label value is automatically set to the most suitable value depending
|
||||
on the used [service discovery type](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs).
|
||||
The `__address__` label can be overriden during relabeling - see [these docs](#how-to-modify-scrape-urls-in-targets).
|
||||
|
||||
Both `instance` and `job` labels can be overriden during relabeling. For example, the following config discovers pod targets
|
||||
in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- target_label: job
|
||||
replacement: foo
|
||||
```
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to modify scrape urls in targets
|
||||
|
||||
URLs for scrape targets are composed of the following parts:
|
||||
|
||||
* Scheme (e.g. `http` or `https`). The scheme is available during target relabeling in a special label - `__scheme__`.
|
||||
By default the scheme is set to `http`. It can be overriden either by specifying the `scheme` option
|
||||
at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) level
|
||||
or by updating the `__scheme__` label during relabeling.
|
||||
* Host and port (e.g. `host12:3456`). This information is available during target relabeling in a special label - `__address__`.
|
||||
Every [supported service discovery type](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs)
|
||||
sets the `__address__` label to the most suitable value. Sometimes this value needs to be modified. In this case
|
||||
just update the `__address__` label during relabeling to the needed value.
|
||||
The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending
|
||||
on the used scheme (`http` or `https`).
|
||||
The final `__address__` label is automatically converted into `instance` label per each target unless the `instance`
|
||||
label is explicitly set during relabeling.
|
||||
* URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `__metrics_path__`.
|
||||
By default the `__metrics_path__` is set to `/metrics`. It can be overriden either by specifying the `metrics_path`
|
||||
option at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
|
||||
or by updating the `__metrics_path__` label during relabeling.
|
||||
* Query args (e.g. `?foo=bar&baz=xyz`). This information is available during target relabeling in special labels
|
||||
with `__param_` prefix. For example, `__param_foo` would have the `bar` value, while `__param_baz` would have the `xyz` value
|
||||
for `?foo=bar&baz=xyz` query string. The query args can be specified either via `params` section
|
||||
at [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
|
||||
or by updating/setting the corresponding `__param_*` labels during relabeling.
|
||||
|
||||
The resulting scrape url looks like the following:
|
||||
|
||||
```
|
||||
<__scheme__> + "://" + <__address__> + <__metrics_path__> + <"?" + query_args_from_param_labels>
|
||||
```
|
||||
|
||||
It is expected that the target exposes metrics
|
||||
in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-based-format)
|
||||
at the resulting scrape url.
|
||||
|
||||
Given the scrape url construction rules above, the following config discovers pod targets
|
||||
in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
|
||||
and constructs per-target scrape url as `https://<pod_name>/foo/bar?baz=<container_name>`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
metrics_path: /foo/bar
|
||||
relabel_configs:
|
||||
- target_label: __scheme__
|
||||
replacement: https
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: __address__
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: __param_baz
|
||||
```
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to copy labels in scrape targets
|
||||
|
||||
Labels can be copied by specifying the source labels via `source_labels` relabeling option
|
||||
and specifying the target label via `target_label` relabeling option.
|
||||
For example, the following config copies `__meta_kubernetes_pod_name` label to `pod` label
|
||||
for all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
```
|
||||
|
||||
Note that the `source_labels` option accepts a list of labels in square brackets. If multiple labels are specified
|
||||
in the `source_labels` list, then the specified label values are joined into a single string with `;` delimiter by default.
|
||||
The delimiter can be modified by specifying it via `separator` option.
|
||||
For example, the following config sets the `pod_name:container_port` value to the `host_port` label
|
||||
for all the discovered pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
|
||||
separator: ":"
|
||||
target_label: host_port
|
||||
```
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to add labels to scrape targets
|
||||
|
||||
Additional labels can be added to scrape targets by specifying the label name in `target_label` relabeling option
|
||||
and by specifying the label value in `replacement` relabeling option.
|
||||
The same approach can be used for updating already existing label values at target level.
|
||||
|
||||
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- target_label: "foo"
|
||||
replacement: "bar"
|
||||
```
|
||||
|
||||
The labels, which are added to the target, are automatically added to all the metrics scraped from the target.
|
||||
For example, if the target exposes the metric `metric{label="value"}`, then the metric is transformed into `metric{label="value",foo="bar"}`
|
||||
before being sent to the storage.
|
||||
|
||||
If the metric exported by the target contains the same label as the target itself, then the `exported_` prefix is added to the exported label name.
|
||||
For example, if the target exposes the metric `metric{foo="baz"}`, then the metric is transformed into `metric{exported_foo="baz",foo="bar"}`.
|
||||
This behaviour can be changed by specifying `honor_labels: true` option at the given scrape config. In this case the exported label overrides
|
||||
the target's label. In this case the `metric{foo="baz"}` stays the same. Example config with `honor_labels: true`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: k8s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
honor_labels: true
|
||||
relabel_configs:
|
||||
- target_label: "foo"
|
||||
replacement: "bar"
|
||||
```
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
## How to drop discovered targets
|
||||
|
||||
Every discovered target contains a set of meta-labels, which start with `__meta_` prefix.
|
||||
The specific sets of labels per each supported service discovery labels are listed
|
||||
[here](https://docs.victoriametrics.com/sd_configs.html#prometheus-service-discovery).
|
||||
|
||||
If a particular discovered target shouldn't be scraped, then `action: keep` or `action: drop` relabeling rules
|
||||
must be used inside `relabel_configs` section.
|
||||
|
||||
The `action: keep` keeps only scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
|
||||
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
|
||||
and scrapes only pods with names starting with `foo` prefix:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: foo_pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- if: '{__meta_kubernetes_pod_name=~"foo.*"}'
|
||||
action: keep
|
||||
```
|
||||
|
||||
The `action: drop` drops all the scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
|
||||
while keeping the rest of targets. For example, the followign config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs)
|
||||
and scrapes only pods with names starting with prefixes other than `foo`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: not_foo_pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- if: '{__meta_kubernetes_pod_name=~"foo.*"}'
|
||||
action: drop
|
||||
```
|
||||
|
||||
See also [useful tips for target relabeling](#useful-tips-for-target-relabeling).
|
||||
|
||||
|
||||
# Useful tips for target relabeling
|
||||
|
||||
* All the labels, which start from `__` prefix, are automatically removed from targets after the relabeling.
|
||||
So it is common practice to store temporary labels with names starting with `__` during target relabeling.
|
||||
* All the target-level labels are automatically added to all the metrics scraped from targets.
|
||||
* The list of discovered scrape targets with all the discovered meta-labels is available at `http://vmagent:8429/service-discovery` page for `vmagent`
|
||||
and at `http://victoriametrics:8428/service-discovery` page for single-node VictoriaMetrics.
|
||||
* The list of active targets with the final set of labels left after relabeling is available at `http://vmagent:8429/targets` page for `vmagent`
|
||||
and at `http://victoriametrics:8428/targets` page for single-node VictoriaMetrics.
|
||||
|
||||
|
||||
## Useful tips for metric relabeling
|
||||
|
||||
* All the labels, which start from `__` prefix, are automatically removed from metrics after the relabeling.
|
||||
So it is common practice to store temporary labels with names startigh with `__` during metrics relabeling.
|
||||
* All the target-level labels are automatically added to all the metrics scraped from targets,
|
||||
so target-level labels are available during metrics relabeling.
|
||||
@ -318,7 +318,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
|
||||
|
||||
## Relabeling
|
||||
|
||||
VictoriaMetrics components (including `vmagent`) support [Prometheus-compatible relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) with [additional enhancements](#relabeling-enhancements) at various stages of data processing. The relabeling can be defined in the following places processed by `vmagent`:
|
||||
VictoriaMetrics components support [Prometheus-compatible relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) with [additional enhancements](#relabeling-enhancements) at various stages of data processing. The relabeling can be defined in the following places processed by `vmagent`:
|
||||
|
||||
* At the `scrape_config -> relabel_configs` section in `-promscrape.config` file. This relabeling is used for modifying labels in discovered targets and for dropping unneded targets. This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
|
||||
* At the `scrape_config -> metric_relabel_configs` section in `-promscrape.config` file. This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics. This relabeling can be debugged by passing `metric_relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs metrics before and after the relabeling and then drops the logged metrics.
|
||||
@ -329,6 +329,7 @@ All the files with relabeling configs can contain special placeholders in the fo
|
||||
|
||||
The following articles contain useful information about Prometheus relabeling:
|
||||
|
||||
* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling.html)
|
||||
* [How to use Relabeling in Prometheus and VictoriaMetrics](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2)
|
||||
* [Life of a label](https://www.robustperception.io/life-of-a-label)
|
||||
* [Discarding targets and timeseries with relabeling](https://www.robustperception.io/relabelling-can-discard-targets-timeseries-and-alerts)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user