From 1be1e9a7a401bbe0c89249d9f425ff1e9b737da0 Mon Sep 17 00:00:00 2001 From: hagen1778 Date: Fri, 24 May 2024 15:09:52 +0200 Subject: [PATCH] deployment/alerts: add new alerting rules `TooLongLabelValues` and `TooLongLabelNames` to notify about truncation of label values or names respectively. Signed-off-by: hagen1778 --- deployment/docker/alerts-health.yml | 22 +++++++++++++++++++++- docs/CHANGELOG.md | 1 + 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/deployment/docker/alerts-health.yml b/deployment/docker/alerts-health.yml index a3f2ea4a8..139860fea 100644 --- a/deployment/docker/alerts-health.yml +++ b/deployment/docker/alerts-health.yml @@ -97,4 +97,24 @@ groups: description: "VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process. For example, `reason=too_long_item` means that time series cannot exceed 64KB. Please, reduce the number of labels or label values for such series. Or enforce these limits via `-maxLabelsPerTimeseries` and - `-maxLabelValueLen` command-line flags." \ No newline at end of file + `-maxLabelValueLen` command-line flags." + + - alert: TooLongLabelValues + expr: increase(vm_too_long_label_values_total[5m]) > 0 + labels: + severity: critical + annotations: + summary: "VictoriaMetrics truncates too long label values" + description: "The maximum length of a label value is limited via `-maxLabelValueLen` cmd-line flag. + Longer label values are truncated and may result into time series overlapping. + Please, check your logs to find which labels were truncated and + either reduce the size of label values or increase `-maxLabelValueLen`". + + - alert: TooLongLabelNames + expr: increase(vm_too_long_label_names_total[5m]) > 0 + labels: + severity: critical + annotations: + summary: "VictoriaMetrics truncates too long label names" + description: "The maximum length of a label name is limited by 256 bytes. + Longer label names are truncated and may result into time series overlapping.". \ No newline at end of file diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index e109bb5a2..1be84519d 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -54,6 +54,7 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/). * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): support reading [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) records in `-datasource.url`, `-remoteWrite.url` and `-remoteRead.url` command-line option. For example, `-remoteWrite.url=http://srv+victoria-metrics` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/), [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): add `-s3TLSInsecureSkipVerify` command-line flag for skipping TLS certificates verification when connecting to S3 endpoint. * FEATURE: expose metric `vm_indexdb_items_dropped_total` to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in `reason` label of the exposed metric. This change also comes with a new [alerting rule](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml) to track changes of this metric. +* FEATURE: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): add new alerting rules `TooLongLabelValues` and `TooLongLabelNames` to notify about truncation of label values or names respectively. * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix bug that prevents the first query trace from expanding on click event. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186). The issue was introduced in [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000) release. * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239).