app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling

For example:

- target_label: composite-label
  replacement: {{source_label1}}-{{source_label2}}

app/vmagent/README.md

@@ -178,7 +178,7 @@ See [the list of supported service discovery types for Prometheus scrape targets
 
 ## scrape_config enhancements
 
-`vmagent` supports the following additional options in `scrape_configs` section:
+`vmagent` supports the following additional options in [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) section:
 
 * `headers` - a list of HTTP headers to send to scrape target with each scrape request. This can be used when the scrape target needs custom authorization and authentication. For example:
 
@@ -199,9 +199,12 @@ scrape_configs:
 * `relabel_debug: true` for enabling debug logging during relabeling of the discovered targets. See [these docs](#relabeling).
 * `metric_relabel_debug: true` for enabling debug logging during relabeling of the scraped metrics. See [these docs](#relabeling).
 
+See [scrape_configs docs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for more details on all the supported options.
+
+
 ## Loading scrape configs from multiple files
 
-`vmagent` supports loading scrape configs from multiple files specified in the `scrape_config_files` section of `-promscrape.config` file. For example, the following `-promscrape.config` instructs `vmagent` loading scrape configs from all the `*.yml` files under `configs` directory, from `single_scrape_config.yml` local file and from `https://config-server/scrape_config.yml` url:
+`vmagent` supports loading [scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) from multiple files specified in the `scrape_config_files` section of `-promscrape.config` file. For example, the following `-promscrape.config` instructs `vmagent` loading scrape configs from all the `*.yml` files under `configs` directory, from `single_scrape_config.yml` local file and from `https://config-server/scrape_config.yml` url:
 
 ```yml
 scrape_config_files:
@@ -210,7 +213,7 @@ scrape_config_files:
 - https://config-server/scrape_config.yml
 ```
 
-Every referred file can contain arbitrary number of [supported scrape configs](#how-to-collect-metrics-in-prometheus-format). There is no need in specifying top-level `scrape_configs` section in these files. For example:
+Every referred file can contain arbitrary number of [supported scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). There is no need in specifying top-level `scrape_configs` section in these files. For example:
 
 ```yml
 - job_name: foo
@@ -338,85 +341,94 @@ The following articles contain useful information about Prometheus relabeling:
 
 ## Relabeling enhancements
 
-VictoriaMetrics provides the following additional relabeling actions on top of standard actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
+* The `replacement` option can refer arbitrary labels via {% raw %}`{{label_name}}`{% endraw %} placeholders. Such placeholders are substituted with the corresponding label value. For example, the following relabeling rule sets `instance-job` label value to `host123-foo` when applied to the metric with `{instance="host123",job="foo"}` labels:
 
-* `replace_all` replaces all of the occurrences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`. For example, the following relabeling config replaces all the occurrences of `-` char in metric names with `_` char (e.g. `foo-bar-baz` metric name is transformed into `foo_bar_baz`):
+  {% raw %}
+  ```yaml
+  - target_label: "instance-job"
+    replacement: "{{instance}}-{{job}}"
+  ```
+  {% endraw %}
+
+* An optional `if` filter can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops metrics, which don't match `foo{bar="baz"}` series selector, while leaving the rest of metrics:
 
   ```yaml
-  - action: replace_all
+  - action: keep
-    source_labels: ["__name__"]
+    if: 'foo{bar="baz"}'
-    target_label: "__name__"
-    regex: "-"
-    replacement: "_"
   ```
 
-* `labelmap_all` replaces all of the occurrences of `regex` in all the label names with the `replacement`. For example, the following relabeling config replaces all the occurrences of `-` char in all the label names with `_` char (e.g. `foo-bar-baz` label name is transformed into `foo_bar_baz`):
+  This is equivalent to less clear Prometheus-compatible relabeling rule:
 
   ```yaml
-  - action: labelmap_all
+  - action: keep
-    regex: "-"
+    source_labels: [__name__, bar]
-    replacement: "_"
+    regex: 'foo;baz'
   ```
 
-* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal, while dropping all the other entries. For example, the following relabeling config keeps targets if they contain equal values for `instance` and `host` labels, while dropping all the other targets:
+* The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
-
-  ```yaml
-  - action: keep_if_equal
-    source_labels: ["instance", "host"]
-  ```
-
-* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal, while keeping all the other entries. For example, the following relabeling config drops targets if they contain equal values for `instance` and `host` labels, while keeping all the other targets:
-
-  ```yaml
-  - action: drop_if_equal
-    source_labels: ["instance", "host"]
-  ```
-
-* `keep_metrics`: keeps all the metrics with names matching the given `regex`, while dropping all the other metrics. For example, the following relabeling config keeps metrics with `fo` and `bar` names, while dropping all the other metrics:
 
   ```yaml
   - action: keep_metrics
-    regex: "foo|bar"
+    regex: "metric_a|metric_b|foo_.+"
   ```
 
-* `drop_metrics`: drops all the metrics with names matching the given `regex`, while keeping all the other metrics. For example, the following relabeling config drops metrics with `foo` and `bar` names, while leaving all the other metrics:
-
   ```yaml
-  - action: drop_metrics
+  - action: keep_metrics
-    regex: "foo|bar"
+    regex:
+    - "metric_a"
+    - "metric_b"
+    - "foo_.+"
   ```
 
-* `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling) for details.
+* VictoriaMetrics provides the following additional relabeling actions on top of standard actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
 
-The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
+  * `replace_all` replaces all of the occurrences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`. For example, the following relabeling config replaces all the occurrences of `-` char in metric names with `_` char (e.g. `foo-bar-baz` metric name is transformed into `foo_bar_baz`):
 
-```yaml
+    ```yaml
-- action: keep_metrics
+    - action: replace_all
-  regex: "metric_a|metric_b|foo_.+"
+      source_labels: ["__name__"]
-```
+      target_label: "__name__"
+      regex: "-"
+      replacement: "_"
+    ```
 
-```yaml
+  * `labelmap_all` replaces all of the occurrences of `regex` in all the label names with the `replacement`. For example, the following relabeling config replaces all the occurrences of `-` char in all the label names with `_` char (e.g. `foo-bar-baz` label name is transformed into `foo_bar_baz`):
-- action: keep_metrics
-  regex:
-  - "metric_a"
-  - "metric_b"
-  - "foo_.+"
-```
 
-VictoriaMetrics components support an optional `if` filter in relabeling configs, which can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops metrics, which don't match `foo{bar="baz"}` series selector, while leaving the rest of metrics:
+    ```yaml
+    - action: labelmap_all
+      regex: "-"
+      replacement: "_"
+    ```
 
-```yaml
+  * `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal, while dropping all the other entries. For example, the following relabeling config keeps targets if they contain equal values for `instance` and `host` labels, while dropping all the other targets:
-- action: keep
-  if: 'foo{bar="baz"}'
-```
 
-This is equivalent to less clear Prometheus-compatible relabeling rule:
+    ```yaml
+    - action: keep_if_equal
+      source_labels: ["instance", "host"]
+    ```
 
-```yaml
+  * `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal, while keeping all the other entries. For example, the following relabeling config drops targets if they contain equal values for `instance` and `host` labels, while keeping all the other targets:
-- action: keep
+
-  source_labels: [__name__, bar]
+    ```yaml
-  regex: 'foo;baz'
+    - action: drop_if_equal
-```
+      source_labels: ["instance", "host"]
+    ```
+
+  * `keep_metrics`: keeps all the metrics with names matching the given `regex`, while dropping all the other metrics. For example, the following relabeling config keeps metrics with `fo` and `bar` names, while dropping all the other metrics:
+
+    ```yaml
+    - action: keep_metrics
+      regex: "foo|bar"
+    ```
+
+  * `drop_metrics`: drops all the metrics with names matching the given `regex`, while keeping all the other metrics. For example, the following relabeling config drops metrics with `foo` and `bar` names, while leaving all the other metrics:
+
+    ```yaml
+    - action: drop_metrics
+      regex: "foo|bar"
+    ```
+
+  * `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling) for details.
 
 ## Graphite relabeling
 

docs/CHANGELOG.md

@@ -24,6 +24,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: return shorter error messages to Grafana and to other clients requesting [/api/v1/query](https://docs.victoriametrics.com/keyConcepts.html#instant-query) and [/api/v1/query_range](https://docs.victoriametrics.com/keyConcepts.html#range-query) endpoints. This should simplify reading these errors by humans. The long error message with full context is still written to logs.
 * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): reduce the amounts of logging at `vmstorage` when `vmselect` connects/disconnects to `vmstorage`.
 * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): improve performance for heavy queries on systems with many CPU cores.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to use {% raw %}`{{label_name}}`{% endraw %} placeholders in the `replacement` option of relabeling rules. This simplifies constructing label values from multiple existing label values. See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling-enhancements) for details.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): generate additional per-target metrics - `scrape_series_limit`, `scrape_series_current` and `scrape_series_limit_samples_dropped` if series limit is set according to [these docs](https://docs.victoriametrics.com/vmagent.html#cardinality-limiter). This simplifies alerting on targets with the exceeded series limit. See [these docs](https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics) for details on these metrics.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for MX record types in [dns_sd_configs](https://docs.victoriametrics.com/sd_configs.html#dns_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/10099).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `__meta_kubernetes_service_port_number` meta-label for `role: service` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) in the same way as Prometheus 2.38 [does](https://github.com/prometheus/prometheus/pull/11002).

docs/vmagent.md

@@ -182,7 +182,7 @@ See [the list of supported service discovery types for Prometheus scrape targets
 
 ## scrape_config enhancements
 
-`vmagent` supports the following additional options in `scrape_configs` section:
+`vmagent` supports the following additional options in [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) section:
 
 * `headers` - a list of HTTP headers to send to scrape target with each scrape request. This can be used when the scrape target needs custom authorization and authentication. For example:
 
@@ -203,9 +203,12 @@ scrape_configs:
 * `relabel_debug: true` for enabling debug logging during relabeling of the discovered targets. See [these docs](#relabeling).
 * `metric_relabel_debug: true` for enabling debug logging during relabeling of the scraped metrics. See [these docs](#relabeling).
 
+See [scrape_configs docs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) for more details on all the supported options.
+
+
 ## Loading scrape configs from multiple files
 
-`vmagent` supports loading scrape configs from multiple files specified in the `scrape_config_files` section of `-promscrape.config` file. For example, the following `-promscrape.config` instructs `vmagent` loading scrape configs from all the `*.yml` files under `configs` directory, from `single_scrape_config.yml` local file and from `https://config-server/scrape_config.yml` url:
+`vmagent` supports loading [scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) from multiple files specified in the `scrape_config_files` section of `-promscrape.config` file. For example, the following `-promscrape.config` instructs `vmagent` loading scrape configs from all the `*.yml` files under `configs` directory, from `single_scrape_config.yml` local file and from `https://config-server/scrape_config.yml` url:
 
 ```yml
 scrape_config_files:
@@ -214,7 +217,7 @@ scrape_config_files:
 - https://config-server/scrape_config.yml
 ```
 
-Every referred file can contain arbitrary number of [supported scrape configs](#how-to-collect-metrics-in-prometheus-format). There is no need in specifying top-level `scrape_configs` section in these files. For example:
+Every referred file can contain arbitrary number of [supported scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). There is no need in specifying top-level `scrape_configs` section in these files. For example:
 
 ```yml
 - job_name: foo
@@ -342,85 +345,94 @@ The following articles contain useful information about Prometheus relabeling:
 
 ## Relabeling enhancements
 
-VictoriaMetrics provides the following additional relabeling actions on top of standard actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
+* The `replacement` option can refer arbitrary labels via {% raw %}`{{label_name}}`{% endraw %} placeholders. Such placeholders are substituted with the corresponding label value. For example, the following relabeling rule sets `instance-job` label value to `host123-foo` when applied to the metric with `{instance="host123",job="foo"}` labels:
 
-* `replace_all` replaces all of the occurrences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`. For example, the following relabeling config replaces all the occurrences of `-` char in metric names with `_` char (e.g. `foo-bar-baz` metric name is transformed into `foo_bar_baz`):
+  {% raw %}
+  ```yaml
+  - target_label: "instance-job"
+    replacement: "{{instance}}-{{job}}"
+  ```
+  {% endraw %}
+
+* An optional `if` filter can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops metrics, which don't match `foo{bar="baz"}` series selector, while leaving the rest of metrics:
 
   ```yaml
-  - action: replace_all
+  - action: keep
-    source_labels: ["__name__"]
+    if: 'foo{bar="baz"}'
-    target_label: "__name__"
-    regex: "-"
-    replacement: "_"
   ```
 
-* `labelmap_all` replaces all of the occurrences of `regex` in all the label names with the `replacement`. For example, the following relabeling config replaces all the occurrences of `-` char in all the label names with `_` char (e.g. `foo-bar-baz` label name is transformed into `foo_bar_baz`):
+  This is equivalent to less clear Prometheus-compatible relabeling rule:
 
   ```yaml
-  - action: labelmap_all
+  - action: keep
-    regex: "-"
+    source_labels: [__name__, bar]
-    replacement: "_"
+    regex: 'foo;baz'
   ```
 
-* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal, while dropping all the other entries. For example, the following relabeling config keeps targets if they contain equal values for `instance` and `host` labels, while dropping all the other targets:
+* The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
-
-  ```yaml
-  - action: keep_if_equal
-    source_labels: ["instance", "host"]
-  ```
-
-* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal, while keeping all the other entries. For example, the following relabeling config drops targets if they contain equal values for `instance` and `host` labels, while keeping all the other targets:
-
-  ```yaml
-  - action: drop_if_equal
-    source_labels: ["instance", "host"]
-  ```
-
-* `keep_metrics`: keeps all the metrics with names matching the given `regex`, while dropping all the other metrics. For example, the following relabeling config keeps metrics with `fo` and `bar` names, while dropping all the other metrics:
 
   ```yaml
   - action: keep_metrics
-    regex: "foo|bar"
+    regex: "metric_a|metric_b|foo_.+"
   ```
 
-* `drop_metrics`: drops all the metrics with names matching the given `regex`, while keeping all the other metrics. For example, the following relabeling config drops metrics with `foo` and `bar` names, while leaving all the other metrics:
-
   ```yaml
-  - action: drop_metrics
+  - action: keep_metrics
-    regex: "foo|bar"
+    regex:
+    - "metric_a"
+    - "metric_b"
+    - "foo_.+"
   ```
 
-* `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling) for details.
+* VictoriaMetrics provides the following additional relabeling actions on top of standard actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
 
-The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
+  * `replace_all` replaces all of the occurrences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`. For example, the following relabeling config replaces all the occurrences of `-` char in metric names with `_` char (e.g. `foo-bar-baz` metric name is transformed into `foo_bar_baz`):
 
-```yaml
+    ```yaml
-- action: keep_metrics
+    - action: replace_all
-  regex: "metric_a|metric_b|foo_.+"
+      source_labels: ["__name__"]
-```
+      target_label: "__name__"
+      regex: "-"
+      replacement: "_"
+    ```
 
-```yaml
+  * `labelmap_all` replaces all of the occurrences of `regex` in all the label names with the `replacement`. For example, the following relabeling config replaces all the occurrences of `-` char in all the label names with `_` char (e.g. `foo-bar-baz` label name is transformed into `foo_bar_baz`):
-- action: keep_metrics
-  regex:
-  - "metric_a"
-  - "metric_b"
-  - "foo_.+"
-```
 
-VictoriaMetrics components support an optional `if` filter in relabeling configs, which can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops metrics, which don't match `foo{bar="baz"}` series selector, while leaving the rest of metrics:
+    ```yaml
+    - action: labelmap_all
+      regex: "-"
+      replacement: "_"
+    ```
 
-```yaml
+  * `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal, while dropping all the other entries. For example, the following relabeling config keeps targets if they contain equal values for `instance` and `host` labels, while dropping all the other targets:
-- action: keep
-  if: 'foo{bar="baz"}'
-```
 
-This is equivalent to less clear Prometheus-compatible relabeling rule:
+    ```yaml
+    - action: keep_if_equal
+      source_labels: ["instance", "host"]
+    ```
 
-```yaml
+  * `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal, while keeping all the other entries. For example, the following relabeling config drops targets if they contain equal values for `instance` and `host` labels, while keeping all the other targets:
-- action: keep
+
-  source_labels: [__name__, bar]
+    ```yaml
-  regex: 'foo;baz'
+    - action: drop_if_equal
-```
+      source_labels: ["instance", "host"]
+    ```
+
+  * `keep_metrics`: keeps all the metrics with names matching the given `regex`, while dropping all the other metrics. For example, the following relabeling config keeps metrics with `fo` and `bar` names, while dropping all the other metrics:
+
+    ```yaml
+    - action: keep_metrics
+      regex: "foo|bar"
+    ```
+
+  * `drop_metrics`: drops all the metrics with names matching the given `regex`, while keeping all the other metrics. For example, the following relabeling config drops metrics with `foo` and `bar` names, while leaving all the other metrics:
+
+    ```yaml
+    - action: drop_metrics
+      regex: "foo|bar"
+    ```
+
+  * `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling) for details.
 
 ## Graphite relabeling
 

lib/promrelabel/config.go

@@ -343,8 +343,10 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
 		graphiteMatchTemplate: graphiteMatchTemplate,
 		graphiteLabelRules:    graphiteLabelRules,
 
-		regexOriginal:                regexOriginalCompiled,
+		regexOriginal: regexOriginalCompiled,
-		hasCaptureGroupInTargetLabel: strings.Contains(targetLabel, "$"),
+
-		hasCaptureGroupInReplacement: strings.Contains(replacement, "$"),
+		hasCaptureGroupInTargetLabel:   strings.Contains(targetLabel, "$"),
+		hasCaptureGroupInReplacement:   strings.Contains(replacement, "$"),
+		hasLabelReferenceInReplacement: strings.Contains(replacement, "{{"),
 	}, nil
 }

lib/promrelabel/if_expression_test.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"encoding/json"
 	"fmt"
+	"strings"
 	"testing"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
@@ -188,6 +189,12 @@ func TestIfExpressionMismatch(t *testing.T) {
 }
 
 func parseMetricWithLabels(metricWithLabels string) ([]prompbmarshal.Label, error) {
+	stripDummyMetric := false
+	if strings.HasPrefix(metricWithLabels, "{") {
+		// Add a dummy metric name, since the parser needs it
+		metricWithLabels = "dummy_metric" + metricWithLabels
+		stripDummyMetric = true
+	}
 	// add a value to metricWithLabels, so it could be parsed by prometheus protocol parser.
 	s := metricWithLabels + " 123"
 	var rows prometheus.Rows
@@ -203,7 +210,7 @@ func parseMetricWithLabels(metricWithLabels string) ([]prompbmarshal.Label, erro
 	}
 	r := rows.Rows[0]
 	var lfs []prompbmarshal.Label
-	if r.Metric != "" {
+	if !stripDummyMetric {
 		lfs = append(lfs, prompbmarshal.Label{
 			Name:  "__name__",
 			Value: r.Metric,

lib/promrelabel/relabel.go

@@ -28,9 +28,11 @@ type parsedRelabelConfig struct {
 	graphiteMatchTemplate *graphiteMatchTemplate
 	graphiteLabelRules    []graphiteLabelRule
 
-	regexOriginal                *regexp.Regexp
+	regexOriginal *regexp.Regexp
-	hasCaptureGroupInTargetLabel bool
+
-	hasCaptureGroupInReplacement bool
+	hasCaptureGroupInTargetLabel   bool
+	hasCaptureGroupInReplacement   bool
+	hasLabelReferenceInReplacement bool
 }
 
 // String returns human-readable representation for prc.
@@ -172,10 +174,16 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
 		return labels
 	case "replace":
 		// Store `replacement` at `target_label` if the `regex` matches `source_labels` joined with `separator`
+		replacement := prc.Replacement
 		bb := relabelBufPool.Get()
+		if prc.hasLabelReferenceInReplacement {
+			// Fill {{labelName}} references in the replacement
+			bb.B = fillLabelReferences(bb.B[:0], replacement, labels[labelsOffset:])
+			replacement = string(bb.B)
+		}
 		bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
 		if prc.Regex == defaultRegexForRelabelConfig && !prc.hasCaptureGroupInTargetLabel {
-			if prc.Replacement == "$1" {
+			if replacement == "$1" {
 				// Fast path for the rule that copies source label values to destination:
 				// - source_labels: [...]
 				//   target_label: foobar
@@ -188,7 +196,7 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
 				// - target_label: foobar
 				//   replacement: something-here
 				relabelBufPool.Put(bb)
-				labels = setLabelValue(labels, labelsOffset, prc.TargetLabel, prc.Replacement)
+				labels = setLabelValue(labels, labelsOffset, prc.TargetLabel, replacement)
 				return labels
 			}
 		}
@@ -203,7 +211,7 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
 		if prc.hasCaptureGroupInTargetLabel {
 			nameStr = prc.expandCaptureGroups(nameStr, sourceStr, match)
 		}
-		valueStr := prc.expandCaptureGroups(prc.Replacement, sourceStr, match)
+		valueStr := prc.expandCaptureGroups(replacement, sourceStr, match)
 		relabelBufPool.Put(bb)
 		return setLabelValue(labels, labelsOffset, nameStr, valueStr)
 	case "replace_all":
@@ -539,3 +547,25 @@ func labelsToString(labels []prompbmarshal.Label) string {
 	b = append(b, '}')
 	return string(b)
 }
+
+func fillLabelReferences(dst []byte, replacement string, labels []prompbmarshal.Label) []byte {
+	s := replacement
+	for len(s) > 0 {
+		n := strings.Index(s, "{{")
+		if n < 0 {
+			return append(dst, s...)
+		}
+		dst = append(dst, s[:n]...)
+		s = s[n+2:]
+		n = strings.Index(s, "}}")
+		if n < 0 {
+			dst = append(dst, "{{"...)
+			return append(dst, s...)
+		}
+		labelName := s[:n]
+		s = s[n+2:]
+		labelValue := GetLabelValueByName(labels, labelName)
+		dst = append(dst, labelValue...)
+	}
+	return dst
+}