diff --git a/CHANGELOG.md b/CHANGELOG.md index 46f61b9..cc414d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,7 @@ The v1.97.x line will be supported for at least 12 months since [v1.97.0](https: * BUGFIX: fix `runtime error: slice bounds out of range` panic, which can occur during query execution. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5733). The bug has been introduced in `v1.97.0`. * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle `avg_over_time({some_filter}[d]) keep_metric_names` queries, where [`some_filter`](https://docs.victoriametrics.com/keyconcepts/#filtering) matches multiple time series with multiple names, while `d` is bigger or equal to `3h`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556). * BUGFIX: dashboards/single: fix typo in query for `version` annotation which falsely produced many version change events. +* BUGFIX: dashboards/all: add new panel `CPU spent on GC`. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed. ## [v1.97.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0) diff --git a/Troubleshooting.md b/Troubleshooting.md index 8c9906a..86fe9ec 100644 --- a/Troubleshooting.md +++ b/Troubleshooting.md @@ -296,6 +296,12 @@ There are the following most commons reasons for slow data ingestion in Victoria which exceeds the interval between ingested samples for the same time series (aka `scrape_interval`). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183) for more details. +1. If you see constant and abnormally high CPU usage of VictoriaMetrics component, try checking `CPU spent on GC` panel + on the corresponding Grafana dasbhoard in `Resource usage` section. If percentage of CPU time spent on garbage collection + is high, then CPU usage of the component can be reduced in cost of memory by changing [GOGC](https://tip.golang.org/doc/gc-guide#GOGC) to higher values. + To change GOGC try setting `GOGC` environment variable to `100` or higher values. By default, VictoriaMetrics components + configured to run with `GOGC=30`. Please note, the higher is GOGC value, the higher will be memory consumption of the component. + ## Slow queries Some queries may take more time and resources (CPU, RAM, network bandwidth) than others.