dashboards: add dashboard and alerts for vmauth (#6491)

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

dashboards/Makefile

@@ -14,3 +14,4 @@ dashboards-sync:
 	SRC=victoriametrics-cluster.json D_UID=oS7Bi_0Wz TITLE="VictoriaMetrics - cluster" $(MAKE) dashboard-copy
 	SRC=vmagent.json D_UID=G7Z9GzMGz TITLE="VictoriaMetrics - vmagent" $(MAKE) dashboard-copy
 	SRC=vmalert.json D_UID=LzldHAVnz TITLE="VictoriaMetrics - vmalert" $(MAKE) dashboard-copy
+	SRC=vmauth.json D_UID=nbuo5Mr4k TITLE="VictoriaMetrics - vmauth" $(MAKE) dashboard-copy

deployment/docker/README.md

@@ -151,8 +151,8 @@ make docker-cluster-vm-datasource-down # shutdown cluster
 ## Alerts
 
 See below a list of recommended alerting rules for various VictoriaMetrics components for running in production. 
-Some of the alerting rules thresholds are just recommendations and could require an adjustment. The list
-of alerting rules is the following:
+Some alerting rules thresholds are just recommendations and could require an adjustment. 
+The list of alerting rules is the following:
 * [alerts-health.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml):
   alerting rules related to all VictoriaMetrics components for tracking their "health" state; 
 * [alerts.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml):
@@ -163,6 +163,8 @@ of alerting rules is the following:
   alerting rules related to [vmagent](https://docs.victoriametrics.com/vmagent/) component;
 * [alerts-vmalert.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmalert.yml):
   alerting rules related to [vmalert](https://docs.victoriametrics.com/vmalert/) component;
+* [alerts-vmauth.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml):
+  alerting rules related to [vmauth](https://docs.victoriametrics.com/vmauth/) component;
 
 Please, also see [how to monitor](https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring) 
 VictoriaMetrics installations.

deployment/docker/alerts-vmauth.yml

@@ -0,0 +1,27 @@
+# File contains default list of alerts for vmauth service.
+# The alerts below are just recommendations and may require some updates
+# and threshold calibration according to every specific setup.
+groups:
+  - name: vmauth
+    interval: 30s
+    rules:
+      - alert: ConcurrentRequestsLimitReached
+        expr: sum(increase(vmauth_concurrent_requests_limit_reached_total[1m])) by (instance) > 0
+        for: 3m
+        labels:
+          severity: warning
+        annotations:
+          summary: "vmauth ({{ $labels.instance }}) reached concurrent requests limit"
+          description: "Possible solutions: increase the limit with flag: -maxConcurrentRequests, 
+          deploy additional vmauth replicas, check requests latency at backend service. 
+          See more details at https://docs.victoriametrics.com/vmauth/#concurrency-limiting"
+      - alert: UserConcurrentRequestsLimitReached
+        expr: sum(increase(vmauth_user_concurrent_requests_limit_reached_total[1m])) by (username) > 0
+        for: 3m
+        labels:
+          severity: warning
+        annotations:
+          summary: "vmauth has reached concurrent requests limit for username {{ $labels.username }}"
+          description: "Possible solutions: increase limit with flag: -maxConcurrentPerUserRequests, 
+          deploy additional vmauth replicas, check requests latency at backend service."
+

deployment/docker/docker-compose-cluster.yml

@@ -33,6 +33,7 @@ services:
       - ./../../dashboards/victoriametrics-cluster.json:/var/lib/grafana/dashboards/vm.json
       - ./../../dashboards/vmagent.json:/var/lib/grafana/dashboards/vmagent.json
       - ./../../dashboards/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
+      - ./../../dashboards/vmauth.json:/var/lib/grafana/dashboards/vmauth.json
 
   # vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
   # where N is number of vmstorages (2 in this case).

deployment/docker/prometheus-cluster.yml

@@ -5,6 +5,9 @@ scrape_configs:
   - job_name: 'vmagent'
     static_configs:
       - targets: ['vmagent:8429']
+  - job_name: 'vmauth'
+    static_configs:
+      - targets: ['vmauth:8427']
   - job_name: 'vmalert'
     static_configs:
       - targets: ['vmalert:8880']

deployment/docker/vm-datasource/docker-compose-cluster.yml

@@ -16,6 +16,7 @@ services:
       - ./../../dashboards/vm/victoriametrics-cluster.json:/var/lib/grafana/dashboards/vm.json
       - ./../../dashboards/vm/vmagent.json:/var/lib/grafana/dashboards/vmagent.json
       - ./../../dashboards/vm/vmalert.json:/var/lib/grafana/dashboards/vmalert.json
+      - ./../../dashboards/vm/vmauth.json:/var/lib/grafana/dashboards/vmauth.json
       - ./vm-datasource/download.sh:/download.sh
     environment:
       - "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=victoriametrics-datasource"

docs/CHANGELOG.md

@@ -29,6 +29,8 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/).
 
 ## tip
 
+* FEATURE: [dashboards](https://grafana.com/orgs/victoriametrics): add [Grafana dashboard](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/vmauth.json) and [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml) for [vmauth](https://docs.victoriametrics.com/vmauth/) dashboard. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4313) for details.
+
 ## [v1.102.0-rc2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0-rc2)
 
 Released at 2024-06-24

docs/vmauth.md

@@ -1020,6 +1020,8 @@ See also [security recommendations](#security).
 
 `vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page
 either via [vmagent](https://docs.victoriametrics.com/vmagent/) or via Prometheus-compatible scraper, so the exported metrics could be analyzed later.
+Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/21394) and [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml)
+for `vmauth` monitoring.
 
 If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass `-metrics.exposeMetadata`
 command-line to them, so they add `TYPE` and `HELP` comments per each exposed metric at `/metrics` page.