Commit Graph

89 Commits

Author SHA1 Message Date
Roman Khavronenko
e8db78eaa4
dashboards: provide copies of Grafana dashboards alternated with Vict… ()
dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-29 11:06:55 +02:00
hagen1778
481a2c70fd
dashboard: fix display of ingested rows rate
Fix display of ingested rows rate for `Samples ingested/s`
and `Samples rate` panels for vmagent's dasbhoard.
Previously, not all ingested protocols were accounted in these panels.
An extra panel `Rows rate` was added to `Ingestion` section to display the split
for rows ingested rate by protocol.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-15 08:45:10 +02:00
hagen1778
d890038a94
dashboards: correctly calculate Bytes per point value
Correctly calculate `Bytes per point` value for single-server and cluster VM dashboards.
Before, the calculation mistakenly accounted for the number of entries in indexdb in
denominator, which could have shown lower values than expected.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-03 16:22:50 +02:00
hagen1778
c47138e1b0
dashboards: add panels for absoulte value of mem and cpu usage by vmalert
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4627

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-03 11:14:14 +02:00
hagen1778
e311a7bf80
dashboards: add Concurrent inserts panel to vmagent's dasbhoard
The new panel supposed to show whether the number of concurrent
inserts processed by vmagent isn't reaching the limit.
The panel contains recommendation what to do if limit is reached.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-03 10:46:25 +02:00
Aliaksandr Valialkin
f35d27aa2b
app/vlstorage: expose vl_data_size_bytes metric at /metrics page for tracking the on-disk data size (both indexdb and the data itself) 2023-07-31 07:56:53 -07:00
Zakhar Bessarab
6f3fee197e
dashboards/cluster: fix using storage filter for cache usage panel ()
Using `job=~$job_storage` forces "Cache usage" panel to display only vmstorage caches, but there is a cache peresent at vmselect(`promql/rollupResult`).
Updated selector to match generic `$job` so that all caches will be displayed with an option to display per-job caches.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-18 11:40:40 +02:00
Artem Navoiev
b024e46284 Add docker compose examples: filebeat(docker, syslog), fluentbit(docker), logstash, vector(docker)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-06-21 03:59:31 -07:00
Roman Khavronenko
ccaa9571ef
Dashboard upd ()
dashboards: update dashboard for single-node version
* add anonymous mem usage panel;
* add syscall rate panel;
* add location to logs panel;
* update legend for panels to reflect instance name;
* update queries to aggregate per instance.

dashboards: update dashboard for cluster version
* add syscall rate panel;
* add drilldown to logs panel.



Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-12 15:58:47 +02:00
Aliaksandr Valialkin
91533531f5
docs/Troubleshooting.md: document an additional case, which could result in slow inserts
If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series,
then vm_slow_row_inserts_total` metric is increased.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
2023-03-20 13:28:36 -07:00
Roman Khavronenko
3eebe52a06
Dashboards upd ()
* dashboards/cluser: use `quantile` since `median` isn't supported by PromQL

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/*: add `restarts` annotation to show when there were restarts

The cluster's annotation query is aggregated `by job`,
while vmagent/vmalert are aggregated `by job, instance`.
This is because cluster dashboard can contains too many instances
and annotation could become too noisy.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/*: support instance filter in Version annotation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-10 17:13:19 +01:00
Roman Khavronenko
2e153b68cd
dashboards: account for indexdb size in Bytes-per-Point panel ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-28 17:47:52 +01:00
Roman Khavronenko
b209d4ace0
dashboards: use median instead of avg ()
`avg` can be affected by just one outlier, which may lead
to false conclusions. `median` is supposed to reflect
reality better by leveling outliers out.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-11 10:01:30 -08:00
Aliaksandr Valialkin
88fed0232c
dashboards: typo fix Datapoints scanned per series -> Datapoints scanned per query 2023-02-03 19:12:33 -08:00
Roman Khavronenko
ec7c3f45ba
dashboards: bump operator dash to v9 of Grafana ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-12 16:31:26 +01:00
Roman Khavronenko
b3a70b8284
dasbhoards: fix the tooltip info for 1.86 ()
See c63755c316 (diff-bba263a473e7fbc9d0fde075ebef6b3d4e32c322ee1210a3e07182292c7723aaR18)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-11 11:30:12 +01:00
Aliaksandr Valialkin
c63755c316
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:20:19 -08:00
Thomas Danielsson
9d1104d812
dashboards: fix operator datasource variable ()
Got "Failed to upgrade legacy queries Datasource $ds was not found" in
Grafana on operator dashboard.
It's datasource variable was incorrectly named `datasource`.

Also made the rest of the dashboards have homogeneous datasource-variable
names and selections, matching vmagent dashboard.
2023-01-05 14:59:56 +01:00
Roman Khavronenko
9d0e1f8e68
dashboards: add backupmanager dashboard ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-04 17:26:15 +01:00
Roman Khavronenko
e40c7d6efa
dashboards: respect $job var in sub-vars for cluster dash ()
Previously, $job_select, $job_storage and $job_insert
didn't respect the $job filter. This change updates
the variable queries to account for set $job variable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 09:53:32 +01:00
Roman Khavronenko
eb275be99d
dashboards: add VersionChange annotation ()
The new annotation is hidden by default and suppose to show
component `short_version` label change on the panels.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-12 16:32:26 +01:00
Roman Khavronenko
0b6b6d52bf
dashboards: remove DataLinks from single version ()
Those data links were copy&paste artifact from cluster version
and aren't needed on the dash.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-07 14:35:52 +01:00
Roman Khavronenko
9f1403db38
dashboards: add non-default flags panel for vmagent ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-07 12:22:20 +01:00
Aliaksandr Valialkin
f3e84b4dea
{dashboards,alerts}: subtitute {type="indexdb"} with {type=~"indexdb.*"} inside queries after 8189770c50
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 16:00:22 -08:00
Roman Khavronenko
6801b37e53
dashboards: add Disk space usage % and Disk space usage % by type panels ()
The new panels have been added to the vmstorage and drilldown rows.

`Disk space usage %` is supposed to show disk space usage percentage.
This panel is now also referred by `DiskRunsOutOfSpace` alerting rule.
This panel has Drilldown option to show absolute values.

`Disk space usage % by type` shows the relation between datapoints
and indexdb size. It supposed to help identify cases when indexdb
starts to take too much disk space.
This panel has Drilldown option to show absolute values.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-05 08:35:33 +01:00
Roman Khavronenko
f989c20dd7
dashboards: fix typo in data link ()
Fixes a missing `&` char in data link for ETA panel
on cluster dashboards. Without `&` char it generates
wrong link when click on Drilldown menu.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 13:21:14 +01:00
Roman Khavronenko
bdd0683c4a
dashboards: update VM single dash ()
The change list is the following:
* bump Grafana version to 9.2.6;
* replace old "Graph" panel with "TimeSeries" panel;
* show % usage of Mem and CPU additionally to of absolute values;
* `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`;
* add Annotations for Alert triggers. Not all alerts are supposed to be displayed
on the dashboard, but only those with label `show_at: dashboard`.
See `alerts.yml` change.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 19:28:22 +01:00
Roman Khavronenko
5d835a6d64
dashboards: update vmalert dash ()
The change list is the following:
* bump Grafana version to 9.2.6;
* replace old Graph panel with TimeSeries panel;
* add RemoteWrite section;
* allow configuring topK elements for some of the panels;
* Preer grouping by job instead of grouping by instance.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 19:26:31 +01:00
Roman Khavronenko
7dfb01bd7b
dashboards: update vmagent dash ()
The change list is the following:
* bump Grafana version to 9.2.6;
* add version change annotations;
* switch to per-job panels instead of per-instance;
* add drilldown option for resource usage panels.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 19:22:13 +01:00
Roman Khavronenko
31ff26065b
dashboards: update VM cluster dash ()
The change list is the following:
* bump Grafana version to 9.2.6;
* remove artifacts in data links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-28 14:13:00 +01:00
Timur Bakeyev
9ad578214e
Update datasource entries consistently contain type prometheus and uid $ds. ()
Co-authored-by: Timour I. Bakeev <tbakeev@ripe.net>
2022-11-28 08:37:39 +01:00
Roman Khavronenko
42e63fe0fd
dashboards: cleanup & remove artifacts ()
* some unexpected DS UIDs were removed;
* replace `$instance.*` filter with `$instance` since we respect
the instance port anyway;
* remove predefined datasource for `clusterbytenant`
in favour of datasource variable `ds`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-25 09:28:14 +01:00
Roman Khavronenko
3407006cdb
dashboards: cluster dashboard update ()
The purpose of the update is to make the dash more usable
for large installations with many instances. Panels which showed
metrics per-instance (Mem, CPU) now are showing metrics per-job or min/max/avg
aggregations in % instead. This supposed to help immediately to identify
resource shortage and remain usable for small and big installations.

For cases when detailed info is needed, to the bottom of the dashboard
a new row `Drilldown` was added. Panels like Mem or CPU now contain
a `data-link` named `Drilldown` (cis shown on line click) which takes
user to more detailed panel.

The change list is the following:
* bump Grafana version to 9.1.0;
* replace old "Graph" panel with "TimeSeries" panel;
* improve Uptime panel to show number of instances per job;
* show % usage of Mem and CPU instead of absolute values;
* `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`;
* add `Drilldown` section for detailed resource usage;
* add Annotations for Alert triggers. Not all alerts are supposed to be displayed
on the dashboard, but only those with label `show_at: dashboard`.
See `alerts-cluster.yml` change.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-23 18:03:25 -08:00
Roman Khavronenko
908fe6a623
dashboards: replace Index size panel with Active series ()
Panel `Index size` showed itself impractical for users. So
replacing it with `Active series` panel.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/776#issuecomment-1255823734
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-25 21:49:18 +02:00
Roman Khavronenko
b4410b1c63
Dashboards ()
* dashboards/cluster: few updates

* apply consistent formatting across panels;
* make resource usage panels per component more detailed;
* add extra panels to vmselect for displaying
`vm_rows_read_per_query`, `vm_rows_scanned_per_query`,
`vm_rows_read_per_series` and `vm_series_read_per_query` metrics.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/single: few updates

* apply consistent formatting across panels;
* add extra panels to Performance for displaying
`vm_rows_read_per_query`, `vm_rows_scanned_per_query`,
`vm_rows_read_per_series` and `vm_series_read_per_query` metrics.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmagent: few updates

* apply consistent formatting across panels;
* add panels for showing number of samples ingested
or scraped;
* adapt resource usage panels for multiple selected jobs/instances;
* add adhoc variable;
* display vmagent's version in Stats.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmalert: few updates

* apply consistent formatting across panels;
* adapt resource usage panels for multiple selected jobs/instances;
* show vmalert version in Stats section.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-16 21:24:32 +02:00
Max Golionko
7da9443686
moved cluster dashboard to master ()
dashboards: move cluster dashboard to master branch

This change should simplify dashboards management.
2022-09-06 16:19:43 +02:00
Roman Khavronenko
289a4862ba
dashboards: add Cache usage % panel to Caches row ()
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2941
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-08 19:37:34 +03:00
Roman Khavronenko
27f1c65074
vmagent: expose metric vmagent_remotewrite_queues ()
The new metric `vmagent_remotewrite_queues` exports a static value of
number of configured remote write queus. This metric is useful to
calculate total saturation per each configured URL with given number
of queues. See corresponding changes to vmagent alerts and dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-18 14:31:35 +03:00
Roman Khavronenko
3960fecac2
dashboards: small visual tweaks for vmagent's dashboard ()
* remove lines filling
* filter series with zero values
* update descriptions

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-05 11:05:35 +02:00
Artem Navoiev
cd7fb05b7c
dashboards: update cluster by tenant dashboard ()
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-06-09 10:39:30 +02:00
Nikolay
cbfc1b7eb8
dashboards: adds dashboard for operator ()
Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Adds proper interval to rate functions
2022-05-23 11:32:51 +03:00
Roman Khavronenko
4c1fbcd6b0
Single dashboards ()
* dashboards: remove index filter from stats panel for DiskUsage

The diskUsage stats panel was showing disk usage without including
size of the index, which is not correct. The filter was removed
to reflect the total disk usage.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2368

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: add adhoc filter to dasbhoard variables

The adhoc filter allows to quickly apply global filters without
modifying the panels.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: add new panel `IndexDB items rate`

The new panel supposed to reflect the pressure on indexDB
caused by churn rate or new series registration.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: rm "Deferred merges" panel since it could be misleading

See more context here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1682#issuecomment-938608067

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: replace fixed interval of `5m` for `rate` expressions

Before we used fixed `5m` interval for expressions with `rate` func.
Unfortunately, this interval wasn't a fit for all the cases. So we
switch to `$__rate_interval` instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: bump version requirement

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: rm `vm_indexdb_items_added_size_bytes_total` expression

Rate over `vm_indexdb_items_added_size_bytes_total` doesn't seem to be useful
on the dasbhoard panel.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-04-24 23:27:56 +03:00
Roman Khavronenko
ea86716d06
dashboards: add row Caches to single node dasbhoard ()
The new row Caches adds more visibility for cache utilization by VM.
It replaces the old `Cache size` panel.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-02-18 13:40:19 +02:00
Roman Khavronenko
445edcc6ac
dashboards: update the threshold for slow inserts % on the dashboard ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-02-15 21:56:53 +02:00
Roman Khavronenko
e29b2b8444
Monitoring single ()
* dashboards: plot cpu limits for vmagent, vmalert and vm-single dashboards

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* alerts: add `TooHighCPUUsage` alert for all VM components

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: bump components version requirements

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-02-15 11:54:28 +02:00
Roman Khavronenko
871528fedb
dashboards/vmagent: fix cached datasource uid ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-12-20 17:32:41 +02:00
Roman Khavronenko
52a3b2d77e
Dashboards vmsingle ()
* dashboards/vmsingle: add "Merges deferred" panel

The new panel supposed to show if there were deferred merges
due to insufficient disk space.
It goes within alerting rule which suppose to send a signal
in such cases.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmsingle: add "Cache usage" panel

The new panel supposed to show the % of the used cache
compared to allowed size by type.
It should help to determine underutilized types of caches.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmsingle: bump version requirement

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmsingle: rm alert for `vm_merge_need_free_disk_space`

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-12-20 17:28:35 +02:00
Roman Khavronenko
bc79bdf68a
Dashboards vmagent updates ()
* dashboards/vmagent: shuffle panels for better visibility

More important error/dropped panels were moved higher on the main row.
Network usage panel moved to Resource usage row.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmagent: add Troubleshooting row to show top 5 instances/jobs by churn rate

New panels are supposed to show top 5 jobs or targets which generate the most
of the churn rate. They were placed into a new row "Troubleshooting".

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmagent: add panels for showing persistent queue saturation

New panels were added to Torubleshooting row to show the persistent queue
saturation. The corresponding alerts were added and linked to these
panels as well.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards/vmagent: add alert "RejectedRemoteWriteDataBlocksAreDropped"

New alert suppose to send a notification when vmagent starts to drop
data blocks rejected by configured remote write destiantion.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-12-20 12:16:53 +02:00
Aliaksandr Valialkin
802f05f73f
dashboards: consistently use regexp filters for template vars ()
Template vars may contain regexp when `all` is selected (.*) or when multiple values are selected (foo|bar).
So they must be passed to regexp filters.
2021-11-09 16:50:21 +02:00
Roman Khavronenko
ea8f625b53
dashboards: add cardnilaity limiter panels for vmagent ()
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-10-18 19:15:33 +03:00