Commit Graph

3823 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
82e38e1627
lib/promscrape: add support for enable_compression option in the same way as Prometheus does
Updates https://github.com/prometheus/prometheus/pull/13166
Updates https://github.com/prometheus/prometheus/issues/12319

Do not document enable_compression option at docs/sd_configs.md, since vmagent already supports
more clear disable_compression option - see https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
2024-02-18 19:42:09 +02:00
Aliaksandr Valialkin
f0db7d474f
lib/promscrape/discovery/kuma: add support for client_id option
See https://github.com/prometheus/prometheus/pull/13278
2024-02-18 19:19:55 +02:00
Aliaksandr Valialkin
690f6c833c
vendor: update github.com/VictoriaMetrics/metricsql from v0.72.1 to v0.73.0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5383
2024-02-18 18:43:41 +02:00
Aliaksandr Valialkin
c1a3876e95
docs/MetricsQL.md: properly document how MetricsQL selects the lookbehind window
- rate(m) isn't equivalent to rate(m[1i]) when step is smaller than the interval between samples.
- default_rollup(m) isn't equivalent to default_rollup(m[1i]) when step is smaller than the interval between samples.

These changes have been made in v1.85.3 as a part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483 .
See the corresponding commit - 9fa3f1dc57 .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5816
2024-02-18 13:48:45 +02:00
Aliaksandr Valialkin
8053ff9135
docs/LTS-releases.md: cosmetic fixes 2024-02-17 18:09:43 +02:00
Aliaksandr Valialkin
55bba932d4
docs/CHANGELOG.md: document f8207e33a2 2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin
cf96c3416d
docs/CHANGELOG.md: add missing for in the description of the TLS configuration features for vmctl
This is a follow-up for 6a07cb1bdb and f973711e56
2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin
efbead081c
docs/Single-server-VictoriaMetrics.md: enumerate all the VictoriaMetrics components 2024-02-17 17:54:34 +02:00
Aliaksandr Valialkin
9f7c9e9033
docs/LTS-releases.md: add a dedicated page describing LTS lines of releases for VictoriaMetrics 2024-02-17 17:53:56 +02:00
hagen1778
e53f53aaf5
app/vmctl: follow-up after 0c293a66ec
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f973711e56)
2024-02-16 15:31:58 +01:00
Khushi Jain
9ce7f21a63
app/vmctl : support TLS config options for remote read mode (#5798)
(cherry picked from commit 0c293a66ec)
2024-02-16 15:31:58 +01:00
hagen1778
025e52adad
app/vmctl: follow-up after 7cd1b7d047
* cleanup code
* update docs

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6a07cb1bdb)
2024-02-16 15:31:58 +01:00
Khushi Jain
02c8b5015c
app/vmctl : support TLS config options for InfluxDB datasource (#5783)
* vmctl: TLS flags for influx DB

* added httputils function

* Add changelog and doc

---------

Co-authored-by: Khushi Jain <khushi.jain@nokia.com>
(cherry picked from commit 7cd1b7d047)
2024-02-16 15:31:57 +01:00
Fred Navruzov
d8de87aeb0
docs: vmanomaly - updates of v1.10.0 and model type section (#5813)
* - apply v1.10 changes
- chapter on model types (uni/multivariate and rolling)

* - update self-monitoring labels description
- fix typos

* fix duplicated text and link rendering

(cherry picked from commit 172e196ac9)
2024-02-16 15:31:56 +01:00
hagen1778
5e4732cc2d
docs: update formatting for usage examples
- Use `sh` format for examples
- Reduce length of progress bars

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3170ad3f44)
2024-02-16 15:31:56 +01:00
Aliaksandr Valialkin
33b2553c78
app/vmstorage: expose vm_last_partition_parts metrics, which may help identifying performance issues related to the increased number of parts in the last partition 2024-02-15 14:52:53 +02:00
Aliaksandr Valialkin
6fd9331b7f
docs/CHANGELOG.md: document v1.93.12 LTS release 2024-02-14 20:17:52 +02:00
Aliaksandr Valialkin
c617379789
docs/CHANGELOG.md: document v1.97.2 LTS release 2024-02-14 18:50:59 +02:00
Aliaksandr Valialkin
77a514f2cc
all: update Docker image tag for VictoriaMetrics components from v1.97.1 to v1.98.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.98.0
2024-02-14 17:14:55 +02:00
Aliaksandr Valialkin
06da06dac0
lib/promrelabel: store the original labels before returning them them to promutils.PutLabels()
This should reduce memory allocations.

This is a follow-up for b09bd6c42a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389
2024-02-14 16:09:38 +02:00
Aliaksandr Valialkin
acc37ff28a
docs/CHANGELOG.md: cut v1.98.0 release 2024-02-14 16:00:59 +02:00
Aliaksandr Valialkin
9df42f000d
docs/CHANGELOG.md: various typo cleanups 2024-02-14 02:45:25 +02:00
Yury Molodov
b08a23c4a5
vmui: improve the context for autocomplete #5736 #5737 #5739 (#5804)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-14 02:36:06 +02:00
Aliaksandr Valialkin
5f2905d120
app/vmselect: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions to MetricsQL
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641
2024-02-13 23:40:30 +02:00
Nikolay
0a2cc0e873
app/vmauth: properly release memory during config reload (#5805)
* app/vmauth: properly release memory during config reload
previously metrics package hold a refrence for channels for users concurrent requests.
it case of churn at `name`  field of users configuration, new metric was created. But previous one wasn't deleted.
It prevented full parsed configuration from being garbace collected.

now all config related metrics are bound to corresponding metrics.Set and unregistered during config reload process.

It also must fix an issue with incorrect values for current concurrent user requests

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-13 20:49:57 +02:00
Aliaksandr Valialkin
89aceb2b2f
docs: add link to https://docs.victoriametrics.com/scrape_config_examples/ to docs about configuring target scraping at vmagent and single-node VictoriaMetrics 2024-02-13 20:48:04 +02:00
Aliaksandr Valialkin
2c44ea89fd
docs/vmbackupmanager.md: mention -license command-line flag instead of deprecated -eula 2024-02-13 20:41:33 +02:00
Aliaksandr Valialkin
3f74cadba4
docs/vmauth.md: add Config reload chapter, which explains how to reload -auth.config at vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194
2024-02-13 20:36:59 +02:00
Aliaksandr Valialkin
f6cb2021c3
docs/enterprise.md: remove the mention of deprecated -eula command-line flag 2024-02-13 18:47:07 +02:00
Aliaksandr Valialkin
dcacf209d9
vendor: update github.com/VictoriaMetrics/metricsql from v0.70.1 to v0.71.0
This adds an ability to propagate label filters across label_set() and alias() functions.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827#issuecomment-1654095358
2024-02-13 06:37:09 +02:00
Aliaksandr Valialkin
67091537ae
app/vmagent/remotewrite: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning tls handshake timeout to -remoteWrite.url
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699
2024-02-13 02:46:24 +02:00
Aliaksandr Valialkin
84e439c469
docs/vmauth.md: add missing dot 2024-02-13 01:09:11 +02:00
Aliaksandr Valialkin
6bc70a883d
app/vmauth: add support for mTLS-based routing of incoming requests to different backends depending on the subject field in the TLS certificate provided by the user
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1547
2024-02-13 01:04:19 +02:00
Aliaksandr Valialkin
ebcbca0e00
docs/keyConcepts.md: do not duplicate the list of supported data ingestion protocols - just refer to the original list at https://docs.victoriametrics.com/#how-to-import-time-series-data
This should simplify keeping docs in sync
2024-02-12 22:55:27 +02:00
Aliaksandr Valialkin
8793817fbe
docs/vmbackup.md: remove the unneeded -storageDataPath command-line from the example for making server-side copy of the backup 2024-02-12 22:24:41 +02:00
Aliaksandr Valialkin
f5680a6857
all: upgrade Go builder from Go1.21.7 to Go1.22.0
See https://go.dev/doc/go1.22
2024-02-12 22:14:00 +02:00
Artem Navoiev
5815ea4160
docs vmbackupmanager update flags
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-02-12 20:16:00 +02:00
Aliaksandr Valialkin
12cf2d8b44
docs/enterprise.md: remove duplicate Enterprise word in the same sentence 2024-02-12 20:13:30 +02:00
Aliaksandr Valialkin
0f46a49912
docs/Single-server-VictoriaMetrics.md: substitute duplicate cases studies list with the link to the original list 2024-02-12 20:13:01 +02:00
Aliaksandr Valialkin
8ea4b8a3b4
docs: remove misleading either from the description of or label filters 2024-02-12 20:12:27 +02:00
Github Actions
c952c678b2
Automatic update operator docs from VictoriaMetrics/operator@e261c37 (#5788) 2024-02-12 20:11:07 +02:00
Roman Khavronenko
433c3726b2
app/vmalert: support filtering for /api/v1/rule like Prometheus does (#5787)
Follow-up after 62e5e2a4c8

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8850c7431d)
2024-02-09 14:36:15 +01:00
Victor Amorim dos Santos
56b1d8e9ed
app/vmalert: support type param for filtering /api/v1/rules response by rule type (#5749)
Co-authored-by: Hui Wang <haley@victoriametrics.com>
(cherry picked from commit 62e5e2a4c8)
2024-02-09 14:36:14 +01:00
Aliaksandr Valialkin
46de5d7eb1
docs: update docs after ae8a867924
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470
2024-02-09 04:20:09 +02:00
Aliaksandr Valialkin
cf64597878
all: add support for specifying multiple -httpListenAddr options 2024-02-09 03:22:49 +02:00
Aliaksandr Valialkin
5bbf95f9b4
docs/CHANGELOG.md: typo fixes 2024-02-08 21:18:52 +02:00
Aliaksandr Valialkin
ae7da12280
lib/httpserver: do not close client connections every 2 minutes by default
Closing client connections every 2 minutes doesn't help load balancing -
this just leads to "jumpy" connections between multiple backend servers,
e.g. the load isn't spread evenly among backend servers, and instead jumps
between the servers every 2 minutes.

It is still possible periodically closing client connections by specifying non-zero -http.connTimeout command-line flag.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037

This is a follow-up for d387da142e
2024-02-08 21:10:54 +02:00
Aliaksandr Valialkin
9216d00a6c
docs/{vmagent,vmalert}: mention that /-/reload endpoint may be protected with -reloadAuthKey command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1172
2024-02-08 18:49:21 +02:00
Aliaksandr Valialkin
0d313dda8a
docs/Cluster-VictoriaMetrics.md: document that /api/v1/query?series_lector[d] returns raw samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148
2024-02-08 18:32:47 +02:00
Aliaksandr Valialkin
fc8328a87f
docs: sync with enterprise branch 2024-02-08 17:25:26 +02:00
Aliaksandr Valialkin
428109a427
docs: update -help output after 83e55456e2 2024-02-08 17:22:42 +02:00
Artem Navoiev
fbe3f51fee
docs: change ndjson links to https://jsonlines.org/ as original one was hacked (#5782)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-02-08 17:22:31 +02:00
hagen1778
2aa86313bc
docs: sync flags description for vmbackup and vmbackupmanager
Sync description after changes in 83e55456e2

Remove extra text in flags description for vmbackupmanager as it breaks layout
when rendered at docs.victoriametrics.com

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-08 17:12:53 +02:00
Aliaksandr Valialkin
44052f4fd5
docs/CHANGELOG.md: add a link to docs describing -disableReroutingOnUnavailable command-line flag 2024-02-08 17:05:27 +02:00
hagen1778
a2063e0e7a
docs: follow-up after 83e55456e2
83e55456e2
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e1926f286b)
2024-02-08 15:58:34 +01:00
Khushi Jain
a076cb4a93
app/vmbackup: support client-side TLS configuration for create/delete snapshot API (#5738)
(cherry picked from commit 83e55456e2)
2024-02-08 15:58:34 +01:00
Aliaksandr Valialkin
d0b55e359a
docs/vmagent.md: add debugging scrape targets chapter 2024-02-08 16:32:54 +02:00
Aliaksandr Valialkin
582d431df2
app/victoria-metrics: properly send staleness markers on victoriametrics shutdown if -selfScrapeInterval > 0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2024-02-08 15:30:40 +02:00
Aliaksandr Valialkin
202d8e2c40
docs: update -help output after 61d9df4c36
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834
2024-02-08 14:50:56 +02:00
Aliaksandr Valialkin
b18e608016
app/vmselect: add ability to reset rollup result cache on startup by passing -search.resetRollupResultCacheOnStartup command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834
2024-02-08 14:42:15 +02:00
Aliaksandr Valialkin
84a3f4719d
docs/CHANGELOG.md: clarify the bugfix description for e1bf8440eb 2024-02-08 13:01:26 +02:00
Aliaksandr Valialkin
ae2a9c8195
lib/mergeset: prevent from possible too big indexBlockSize panic
This panic could occur when samples with too long label values are ingested into VictoriaMetrics.
This could result in too long fistItem and commonPrefix values at blockHeader (up to 64kb each).
This may inflate the maximum index block size by 4 * maxIndexBlockSize.
2024-02-08 12:55:58 +02:00
hagen1778
9b173c2f01
dashboards: follow-up 4369bc1df2
* add more details to changelog
* simplify panels description
* remove capacity planning recommendation, as it proves it incompetent

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-08 12:55:42 +02:00
Hui Wang
0cd0ddc1c1
deployment/dashboards: fix Storage full ETA panels (#5747)
During background downsampling, rate(vm_deduplicated_samples_total{type="merge"}) could be much bigger than 
rate(vm_rows_added_to_storage_total) and it could last quite some time,
 which causes negative values of Storage full ETA and confuses users, see playground.

Instead of trying to get more accurate results during downsampling, I think it's ok to ignore 
vm_deduplicated_samples_total at all, it's more reasonable to see Storage full ETA increase after downsampling.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-08 12:54:31 +02:00
Aliaksandr Valialkin
def5573f92
app/vmselect/promql: properly handle precision errors in rollup functions
changes(), increases_over_time() and resets() shouldn't take into account
value changes, which may occur because of precision errors.

The maximum guaranteed precision for raw samples stored in VictoriaMetrics is 12 decimal digits.
So do not count relative changes for values if they are smaller than 1e-12 comparing to the value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767
2024-02-08 02:33:17 +02:00
Aliaksandr Valialkin
f4252cd9aa
docs/CHANGELOG.md: properly document the change at b74006e2ca
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774
2024-02-07 22:06:37 +02:00
Nihal
95287d269b
[vmsingle/vminsert]: change http success response code to 200 for -/reload request handler (#5776)
* change vmsingle's response code to 200 for reload request handler

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* change vmsingle's response code to 200 for the reload request handler

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* change vmsingle's response code to 200 for the reload request handler. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

---------

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
2024-02-07 22:06:24 +02:00
Github Actions
3c069607b3
Automatic update operator docs from VictoriaMetrics/operator@38dfad5 (#5772) 2024-02-07 21:55:11 +02:00
Aliaksandr Valialkin
5d9e0ab71e
docs/CHANGELOG.md: support empty command-line flag values in short array notation
For example, -fooDuration=',10s,' is now supported - it sets three command-line flag values:

- the first and the last one are set to the default value for `-fooDuration`
- the second one is set to 10s
2024-02-07 20:55:01 +02:00
Aliaksandr Valialkin
949481fe48
docs/relabeling.md: add examples on how to rename scraped metrics and how to add/change labels in scraped metrics 2024-02-07 20:29:04 +02:00
Aliaksandr Valialkin
caf706fcc0
all: update Go builder from Go1.21.6 to Go1.21.7
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved
2024-02-07 04:01:05 +02:00
Aliaksandr Valialkin
18e1dfd7a8
docs/scrape_config_examples.md: add an example for scraping cadvisor in Kubernetes 2024-02-07 03:23:01 +02:00
Aliaksandr Valialkin
898414856f
docs/Cluster-VictoriaMetrics.md: document the /datadog/api/beta/sketches endpoint
This is a follow-up for a1d1ccd6f2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584
2024-02-07 01:36:08 +02:00
Aliaksandr Valialkin
82f4e4e070
app/{vmagent,vminsert}: follow-up after a1d1ccd6f2
- Document the change at docs/CHANGELOG.md
- Copy changes from docs/Single-server-VictoriaMetrics.md to README.md
- Add missing handler for processing multitenant requests ( https://docs.victoriametrics.com/vmagent/#multitenancy )
- Substitute github.com/stretchr/testify dependency with 3 lines of code in the added tests
- Comment unclear code at lib/protoparser/datadogsketches/parser.go , so @AndrewChubatiuk could update it
  and add permalinks to the original source code there.
- Various code cleanups

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3091
2024-02-07 01:31:52 +02:00
Daria Karavaieva
0ff3e6f3bc
Vmanomaly Guide - dashboard and query change (#5771)
* dashboard fix

* query fix

* changed screenshots

* minor fixes
2024-02-07 01:31:15 +02:00
hagen1778
eb90f2170a
docs/vmalert: mention step param when comapring rule results to raw queries
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-07 01:30:32 +02:00
Andrii Chubatiuk
c634859c4f
support datadog /api/beta/sketches API (#5584)
Co-authored-by: Andrew Chubatiuk <andrew.chubatiuk@motional.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-07 01:30:00 +02:00
Yury Molodov
5778acf9eb
vmui: improve select component functionality (#5755)
* vmui: fix select closing on click outside (#5728)

* vmui: clear entered text in select after selecting a value (#5727)

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-06 22:50:28 +02:00
Yury Molodov
0b5f5d456c
vmui: fix handling invalid timezone (#5758)
* vmui: fix handling invalid timezone (#5732)

* vmui: switch browser timezone flag to isValid

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-06 22:48:00 +02:00
Yury Molodov
0cf17068b8
vmui: fix graph dragging (#5769)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-06 22:32:43 +02:00
Aliaksandr Valialkin
ebd0f1b72e
vendor: update github.com/VictoriaMetrics/metricsql from v0.70.0 to v0.70.1
This should help with the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604
2024-02-06 21:52:55 +02:00
Aliaksandr Valialkin
dc340a83a9
docs/scrape_config_examples.md: document that full urls can be used as in the targets list 2024-02-06 19:49:23 +02:00
Aliaksandr Valialkin
2fec67632f
docs/scrape_config_examples.md: add missing to specify phrase 2024-02-06 19:43:30 +02:00
Aliaksandr Valialkin
0504771db1
docs/CHANGELOG.md: add link to mTLS feature request 2024-02-06 19:32:51 +02:00
Aliaksandr Valialkin
bace92fab6
lib/httpserver: add support for mTLS for requests to -httpListenAddr 2024-02-06 17:47:27 +02:00
Aliaksandr Valialkin
79990cc89b
docs/CHANGELOG.md: move descriptions for recently added features from v1.97.1 to tip 2024-02-06 16:31:44 +02:00
Aliaksandr Valialkin
22524353ed
docs/scrape_config_examples.md: add examples for typical scrape_config usage 2024-02-06 16:01:34 +02:00
Aliaksandr Valialkin
01312003be
docs/Cluster-VictoriaMetrics.md: add a warning that -disableReroutingOnUnavailable command-line flag may result in long pause for data ingestion when some of vmstorage nodes are unavailable for long time 2024-02-06 16:01:07 +02:00
Aliaksandr Valialkin
bffcb40537
docs/enterprise.md: prioritize enterprise technical support over other enterprise features 2024-02-06 16:00:21 +02:00
hagen1778
8bc18b5a4c
docs/vmalert: update limit description
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-06 16:00:02 +02:00
hagen1778
702cc4514a
docs/vmalert: mention limit option in group params
This param was supported for long time but was missing in the docs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-06 15:59:28 +02:00
Fred Navruzov
eaf75a565f
update changelog with 1.0.0 helm chart ref (#5763) 2024-02-06 15:58:49 +02:00
Aliaksandr Valialkin
2bad79a97e
docs/sd_configs.md: improve readability a bit 2024-02-05 15:33:23 +02:00
Github Actions
c0b2ccb6ca
Automatic update operator docs from VictoriaMetrics/operator@bfc521d (#5761)
Co-authored-by: Hui Wang <haley@victoriametrics.com>
2024-02-05 15:33:22 +02:00
Aliaksandr Valialkin
63a43331a3
docs/Cluster-VictoriaMetrics.md: document -disableReroutingOnUnavailable command-line flag
This is a follow-up for 88f0d1572e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713
2024-02-05 15:17:09 +02:00
Aliaksandr Valialkin
67d166f181
docs/Cluster-VictoriaMetrics.md: move the Improve re-routing performance during restart to more appropriate place
Previoulsy it was mistakenly inserted between `No downtime strategy` and 'Minimum downtime strategy' chapters.

This is a follow-up for 37997abd14
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5293
2024-02-05 14:44:06 +02:00
Aliaksandr Valialkin
e2c58a9db1
docs/Cluster-VictoriaMetrics.md: mention -metrics.exposeMetadata command-line flag in Monitoring section 2024-02-05 11:48:05 +02:00
Aliaksandr Valialkin
103f527037
docs/CHANGELOG.md: add a link to all VictoriaMetrics dashboards for Grafana 2024-02-05 11:45:39 +02:00
Aliaksandr Valialkin
6af029fd9b
docs/Troubleshooting.md: clarify the section about GOGC tuning
This is a follow-up for 487a94565b
2024-02-05 11:45:39 +02:00
Aliaksandr Valialkin
b105a5a1f5
docs/CHANGELOG.md: add missing links to the corresponding dashboards 2024-02-05 11:45:06 +02:00
Aliaksandr Valialkin
19f9203c7a
docs: mention -metrics.exposeMetadata command-line flag in Monitoring sections
This is a follow-up for 326a77c697
2024-02-05 11:44:39 +02:00
Artem Navoiev
e3138a356c
docs: fix aliasis for url-example
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-02-05 11:43:58 +02:00
Fred Navruzov
be41ffb925
docs: vmanomaly - fix 1.9.2 references in pull cmd and docs (#5756)
* fix 1.9.2 references in pull cmd and docs
* better readability of upgrade note
2024-02-05 11:43:41 +02:00
hagen1778
bdbab7bed5
dashboards/all: add new panel CPU spent on GC
It should help identifying cases when too much CPU is spent on garbage collection,
 and advice users on how this can be addressed.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-05 11:42:28 +02:00
hagen1778
2206309439
dashboards: add Targets scraped/s
A new stat panel shows the number of targets scraped by the vmagent per-second.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-05 10:51:35 +02:00
Aliaksandr Valialkin
9f23231926
docs/CHANGELOG.md: document v1.93.11 LTS release 2024-02-01 18:21:35 +02:00
Aliaksandr Valialkin
d63bc81f29
docs/CHANGELOG.md: document v1.87.14 LTS release 2024-02-01 17:09:03 +02:00
Aliaksandr Valialkin
24437b6d70
all: update VictoriaMetrics Docker image from v1.97.0 to v1.97.1
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.1
2024-02-01 16:15:14 +02:00
Aliaksandr Valialkin
dab3da9ee8
docs: mark v1.97.x as long-term support release 2024-02-01 15:16:52 +02:00
Aliaksandr Valialkin
e1ceb64432
docs/CHANGELOG.md: cut v1.97.1 2024-02-01 15:08:53 +02:00
Aliaksandr Valialkin
eff4dc3685
docs: sync -help output after recent changes 2024-02-01 15:06:19 +02:00
Aliaksandr Valialkin
d8f508cc21
docs/Single-server-VictoriaMetrics.md: clarify Security chapter a bit 2024-02-01 14:45:10 +02:00
Aliaksandr Valialkin
ee327cfd5d
docs/Single-server-VictoriaMetrics.md: run make docs-sync after 49d5e7fef5
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5262
2024-02-01 14:42:07 +02:00
Aliaksandr Valialkin
6c56f49f9c
lib/prompbmarshal: return back custom protobuf marshaler for lib/prompbmarshal.WriteRequest
The easyproto-based marshaler is 2x slower than the previous custom marshaler,
so let's stick with it. This improves the performance for sending data to remote storage at vmagent
and reduces CPU usage to pre-v1.97.0 levels.
2024-02-01 06:34:46 +02:00
Aliaksandr Valialkin
cc626ae3b5
lib/promauth: follow-up for fca3b14b7b
- Simplify the code for handling BasicAuthConfig at lib/promauth/config.go
- Move the description of the change into correct place at docs/CHANGELOG.md
- Put tests for username in front of tests for password at lib/promauth/config_test.go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5720
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511
2024-01-31 19:47:53 +02:00
Nihal
bcd094ac8b
Support for username_file in scrape config (basic_auth) similar to Prometheus for having config compatibility (#5720)
* adding support for username_file in basic_auth of scrape config

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* adding support for username_file in basic_auth of scrape config. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* adding support for username_file in basic_auth of scrape config

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* adding support for username_file in basic_auth of scrape config

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

* adding support for username_file in basic_auth of scrape config

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>

---------

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
2024-01-31 19:47:50 +02:00
Aliaksandr Valialkin
4f5cb17042
app/vmselect/netstorage: properly handle the case when an empty brsPool points to the end of brs.brs
This case is possible after a new brsPool is allocated. The fix is to verify whether len(brsPool) >= len(brs.brs)
before trying to append a new item to brsPool and sharing its contents with brs.brs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5733
2024-01-31 10:31:51 +02:00
hagen1778
151247c9b9
dashboards/single: fix typo in query for version annotation
The typo falsely produced many version change events.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-31 10:28:24 +02:00
Aliaksandr Valialkin
2033fe4caf
app/vmselect/promql: really keep metric names when keep_metric_names modifier is applied to binary operator
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556
2024-01-31 02:33:06 +02:00
Aliaksandr Valialkin
a0043e214f
docs/vmauth.md: typo fix after 68be182075 2024-01-31 00:13:56 +02:00
Aliaksandr Valialkin
f1effc1133
all: update VictoriaMetrics Docker image from v1.96.0 to v1.97.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0
2024-01-30 23:37:42 +02:00
Aliaksandr Valialkin
6d766abde7
deployment/docker: upgrade base Docker image from Alpine 3.19.0 to 3.19.1
See https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html
2024-01-30 22:47:29 +02:00
Aliaksandr Valialkin
966f1ad07f
docs/CHANGELOG.md: cut v1.97.0 2024-01-30 22:45:10 +02:00
Dan Dascalescu
d2c38e2588
docs: t...over_time functions return fractional seconds (#5715)
* docs: t...over_time functions return fractional seconds

* Apply suggestions from code review

---------

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-30 22:20:16 +02:00
Roman Khavronenko
02e609b141
app/vmselect: set proper timestamp for cached instant responses (#5723)
* app/vmselect: set proper timestamp for cached instant responses

The change updates `getSumInstantValues` to prefer timestamp
from the most recent results. Before, timestamp from cached series
was used.

The old behavior had negative impact on recording rules as they
were getting responses with shifted timestamps in past.
Subsequent recording or alerting rules fetching results of these
recording rules could get no result due to staleness interval.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-30 22:20:16 +02:00
Yury Molodov
587c5fec2d
vmui: add the ability to expand all tracing entries (#5677) (#5726) 2024-01-30 21:13:13 +02:00
Github Actions
317a2551ee
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@40e4e15 (#5729) 2024-01-30 21:07:50 +02:00
Aliaksandr Valialkin
1a071c5228
docs/CHANGELOG.md: document the enhancement, which reduces initial memory usage when vmagent scrapes targets with large responses
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567
2024-01-30 20:51:29 +02:00
Aliaksandr Valialkin
dc9f4f0907
docs/CHANGELOG.md: refer to the related pull request for the bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945 2024-01-30 20:21:51 +02:00
Aliaksandr Valialkin
16787415b3
docs/CHANGELOG.md: document the bugfix addressed by the commit bc7cf4950b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945
2024-01-30 20:16:32 +02:00
Artem Navoiev
17a4d310d0
docs: add alias for keyconcepts
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-30 18:39:59 +02:00
Aliaksandr Valialkin
09c388a8e4
lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode
While fasthttp.Client uses less CPU and RAM when scraping targets with small responses (up to 10K metrics),
it doesn't work well when scraping targets with big responses such as kube-state-metrics.
In this case it could use big amounts of additional memory comparing to net/http.Client,
since fasthttp.Client reads the full response in memory and then tries re-using the large buffer
for further scrapes.

Additionally, fasthttp.Client-based scraping had various issues with proxying, redirects
and scrape timeouts like the following ones:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5425
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017

This should help reducing memory usage for the case when target returns big response
and this response is scraped by fasthttp.Client at first before switching to stream parsing mode
for subsequent scrapes. Now the switch to stream parsing mode is performed on the first scrape
after reading the response body in memory and noticing that its size exceeds the value passed
to -promscrape.minResponseSizeForStreamParse command-line flag.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567

Overrides https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931
2024-01-30 18:39:55 +02:00
Aliaksandr Valialkin
36be93f53c
docs/keyConcepts.md: clarify the information about which data is returned by instant and range queries
Do not use `raw samples` term there, since it adds more confusion than clarity:
the `raw samples` refers to real samples stored in the database, while neither range nor instant queries
do not return raw samples - they both return *calculated* samples at *the given* timestamps.

This is a follow-up for b5978ed8f9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5710
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5708
2024-01-29 18:22:27 +01:00
Fred Navruzov
ff67f4d3fb
- fix link/version of helm chart in update request (#5716) 2024-01-29 18:22:11 +01:00
Artem Navoiev
00db646851
docs: add backward compaitble redicrt for url examples page
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-29 18:21:03 +01:00
Fred Navruzov
8043cf35d8
- update versions to 1.9.2 (#5714)
- update guide asset urls to flat
2024-01-29 18:20:38 +01:00
hagen1778
131afea8ab
docs: specify results of Instant and Range queries
Mention explicitly what are value and timestamp field in returned
results from Instant and Range queries.

Updates
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5710
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5708

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 18:20:02 +01:00
Fred Navruzov
790ee9af6d
update guide and changelog to 1.9.1 (#5706) 2024-01-29 17:23:06 +01:00
hagen1778
216f268c1a
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 17:06:26 +01:00
hagen1778
3baeface89
docs: follow-up after 491287ed15
491287ed15
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 16:59:13 +01:00
Artem Navoiev
0d210308c3
docs: remove witdh from images, remove <p>, remove <div> (#5705)
* docs: remove witdh from images, remove <p>, remove <div>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* docs: remove <div> clarify language in code blocks

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-29 16:50:51 +01:00
Daria Karavaieva
469bbbb2fa
version 1.9.1 update, dashboard viz flag (#5704) 2024-01-29 16:48:55 +01:00
Aliaksandr Valialkin
f5559c038c
lib/storage: do not check the limit for -search.maxUniqueTimeseries when performing /api/v1/labels and /api/v1/label/.../values requests
This limit has little sense for these APIs, since:

- Thses APIs frequently result in scanning of all the time series on the given time range.
  For example, if extra_filters={datacenter="some_dc"} .

- Users expect these APIs shouldn't hit the -search.maxUniqueTimeseries limit,
  which is intended for limiting resource usage at /api/v1/query and /api/v1/query_range requests.

Also limit the concurrency for /api/v1/labels, /api/v1/label/.../values
and /api/v1/series requests in order to limit the maximum memory usage and CPU usage for these API.
This limit shouldn't affect typical use cases for these APIs:

- Grafana dashboard load when dashboard labels should be loaded
- Auto-suggestion list load when editing the query in Grafana or vmui

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055
2024-01-29 16:44:46 +01:00
Roman Khavronenko
9e9f170fe7
lib/streamaggr: skip unfinished aggregation state on shutdown by default (#5689)
Sending unfinished aggregate states tend to produce unexpected anomalies with lower values than expected.
The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 22:45:45 +01:00
Roman Khavronenko
562edb72ea
app/vmalert: fix data race during hot-config reload (#5698)
* app/vmalert: fix data race during hot-config reload

During hot-reload, the logic evokes the group update and rules evaluation
interruption simultaneously. Falsely assuming that interruption happens before
the update. However, it could happen that group will be updated first and only
after the rules evaluation will be cancelled. Which will result in permanent
interruption for all rules within the group.

The fix caches the cancel context function into local variable first. And only after
performs the group update. With cached cancel function we can safely call it without
worrying that we cancel the evaluation for already updated group.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "app/vmalert: fix data race during hot-config reload"

This reverts commit a4bb7e8932.

* app/vmalert: fix data race during hot-config reload

During hot-reload, the logic evokes the group update and rules evaluation
interruption simultaneously. Falsely assuming that interruption happens before
the update. However, it could happen that group will be updated first and only
after the rules evaluation will be cancelled. Which will result in permanent
interruption for all rules within the group.

The fix cancels the evaulation context before applying the update, making sure
that the context will be cancelled for old group always.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 22:43:02 +01:00
Yury Molodov
551f48466c
vmui: fix Enter key in query field (#5667) (#5681) 2024-01-26 22:38:51 +01:00
Artem Navoiev
d42908133c
docs: remove <p> for imanges (#5702)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 22:34:40 +01:00
Artem Navoiev
36fa314161
remove all <div> as far they obsolete and can break markdown (#5701)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 22:32:54 +01:00
Aliaksandr Valialkin
2b0123058a
docs: update -help output after bb7a419cc3 2024-01-26 22:29:22 +01:00
Aliaksandr Valialkin
7a8b92b590
lib/{mergeset,storage}: make background merge more responsive and scalable
- Maintain a separate worker pool per each part type (in-memory, file, big and small).
  Previously a shared pool was used for merging all the part types.
  A single merge worker could merge parts with mixed types at once. For example,
  it could merge simultaneously an in-memory part plus a big file part.
  Such a merge could take hours for big file part. During the duration of this merge
  the in-memory part was pinned in memory and couldn't be persisted to disk
  under the configured -inmemoryDataFlushInterval .

  Another common issue, which could happen when parts with mixed types are merged,
  is uncontrolled growth of in-memory parts or small parts when all the merge workers
  were busy with merging big files. Such growth could lead to significant performance
  degradataion for queries, since every query needs to check ever growing list of parts.
  This could also slow down the registration of new time series, since VictoriaMetrics
  searches for the internal series_id in the indexdb for every new time series.

  The third issue is graceful shutdown duration, which could be very long when a background
  merge is running on in-memory parts plus big file parts. This merge couldn't be interrupted,
  since it merges in-memory parts.

  A separate pool of merge workers per every part type elegantly resolves both issues:
  - In-memory parts are merged to file-based parts in a timely manner, since the maximum
    size of in-memory parts is limited.
  - Long-running merges for big parts do not block merges for in-memory parts and small parts.
  - Graceful shutdown duration is now limited by the time needed for flushing in-memory parts to files.
    Merging for file parts is instantly canceled on graceful shutdown now.

- Deprecate -smallMergeConcurrency command-line flag, since the new background merge algorithm
  should automatically self-tune according to the number of available CPU cores.

- Deprecate -finalMergeDelay command-line flag, since it wasn't working correctly.
  It is better to run forced merge when needed - https://docs.victoriametrics.com/#forced-merge

- Tune the number of shards for pending rows and items before the data goes to in-memory parts
  and becomes visible for search. This improves the maximum data ingestion rate and the maximum rate
  for registration of new time series. This should reduce the duration of data ingestion slowdown
  in VictoriaMetrics cluster on e.g. re-routing events, when some of vmstorage nodes become temporarily
  unavailable.

- Prevent from possible "sync: WaitGroup misuse" panic on graceful shutdown.

This is a follow-up for fa566c68a6 .
Thanks @misutoth to for the inspiration at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2024-01-26 22:19:52 +01:00
Artem Navoiev
9e0416c666
docs: delete docs/provision_datasources.png as we support webp
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:36:19 +01:00
Github Actions
41c48a4c59
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@ef5cfe6 (#5700) 2024-01-26 21:35:53 +01:00
Aliaksandr Valialkin
9e70d9ab47
docs/Makefile: mention that the Makefile rules must be run from VictoriaMetrics repository root 2024-01-26 21:11:14 +01:00
Artem Navoiev
2b870b1116
docs: fix key concepts image and links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:10:36 +01:00
Artem Navoiev
bb5a0719a5
docs: change [image] to img as far we support it in release guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:09:22 +01:00
Artem Navoiev
89fbefefdb
docs: remoev vmanomaly as far we have dedicated section with alredy exists redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:08:54 +01:00
Artem Navoiev
6ed9a05a08
docs: vmanomaly fix images
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:08:28 +01:00
Daria Karavaieva
b042982339
Vmanomaly Guide dashboard provisioning (#5679)
* dashboard provisioning

* delete dashboard filter, new query

* dashboard screens, guide fixes
2024-01-26 21:07:44 +01:00
Artem Navoiev
aee3e51315
docs: remove raw and endraw tags as they are not needed for the new v… (#5696)
* docs: remove raw and endraw tags as they are not needed for the new version of site

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* revert formating in vmaler

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 21:06:10 +01:00
Github Actions
cd287b2e4c
Automatic update operator docs from VictoriaMetrics/operator@0628def (#5694) 2024-01-26 21:05:13 +01:00
Github Actions
b8a4a78fef
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@c644bec (#5691) 2024-01-26 20:52:47 +01:00
Roman Khavronenko
a2f83115ae
app/vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0 (#5680)
* app/vmalert: autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0`

 Previously, `ALERTS_FOR_STATE` was generated only for alerts with `for > 0`.
 This behavior differs from Prometheus behavior - it generates ALERTS_FOR_STATE
 time series for alerting rules with `for: 0` as well. Such time series can
 be useful for tracking the moment when alerting rule became active.

 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5648
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3056

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmalert: support ALERTS_FOR_STATE in `replay` mode

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 20:51:50 +01:00
Github Actions
362e52f880
Automatic update operator docs from VictoriaMetrics/operator@e75a096 (#5690) 2024-01-26 20:50:57 +01:00
Github Actions
8fa6c42396
Automatic update operator docs from VictoriaMetrics/operator@f6b9c08 (#5676) 2024-01-26 20:50:00 +01:00
hagen1778
bd7ebb41b2
docs: fix the issue link
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 20:49:01 +01:00
Alexander Marshalov
14712e3b99
vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached. (#5682) 2024-01-25 10:21:09 +02:00
Aliaksandr Valialkin
1e364c992d
lib/promscrape/discovery/kubernetes: do not generate targets for already terminated pods and containers
Already terminated pods and containers cannot be scraped and will never resurrect,
so there is zero sense in creating scrape targets for them.
2024-01-24 14:58:51 +02:00
Aliaksandr Valialkin
0dca3c4025
app/{vmselect,vmstorage}: return compression of the data passed from vmstorage to vmselect
This reverts cd4f641d32 , since it has been appeared that the disabled compression
for vmstorage->vmselect data increase network bandwidth usage by more than 10x on typical production workloads,
while it decreases CPU usage at vmstorage by up to 10% and improves query latency by up to 10%.

The 10x increase in network usage is too high price for 10% improvements on query latency and vmstorage CPU usage.
This may result in network bandwidth bottlenecks, which can reduce the overall performance and stability
of VictoriaMetrics cluster. That's why return back the vmstorage->vmselect data compression by default.

The vmstorage->vmselect compression can be disabled by passing -rpc.disableCompression command-line flag to vmstorage.
The vmselect->vmselect compression in multi-level cluster setup can be disabled by passing -clusternative.disableCompression command-line flag.
2024-01-24 13:37:05 +02:00
Aliaksandr Valialkin
e6e5b97e1e
lib/streamaggr: expand %{ENV} placeholders in stream aggregation configs 2024-01-24 12:31:42 +02:00
Aliaksandr Valialkin
12698b9136
lib/mergeset: really limit the number of in-memory parts to 15
It has been appeared that the registration of new time series slows down linearly
with the number of indexdb parts, since VictoriaMetrics needs to check every indexdb part
when it searches for TSID by newly ingested metric name.

The number of in-memory parts grows when new time series are registered
at high rate. The number of in-memory parts grows faster on systems with big number
of CPU cores, because the mergeset maintains per-CPU buffers with newly added entries
for the indexdb, and every such entry is transformed eventually into a separate in-memory part.

The solution has been suggested in https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212
by @misutoth - to limit the number of in-memory parts with buffered channel.
This solution is implemented in this commit. Additionally, this commit merges per-CPU parts
into a single part before adding it to the list of in-memory parts. This reduces CPU load
when searching for TSID by newly ingested metric name.

The https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 recommends setting the limit on the number
of in-memory parts to 100, but my internal testing shows that much lower limit 15 works with the same efficiency
on a system with 16 CPU cores while reducing memory usage for `indexdb/dataBlocks` cache by up to 50%.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190
2024-01-24 03:41:19 +02:00
Aliaksandr Valialkin
5205f1c6a6
docs/Cluster-VictoriaMetrics.md: document that vmstorage doesnt compress data it sends to vmselect by default
This is a follow-up for cd4f641d32
2024-01-23 23:21:31 +02:00
Github Actions
4ccf3f41c6
Automatic update operator docs from VictoriaMetrics/operator@1470569 (#5668) 2024-01-23 17:47:36 +02:00
Aliaksandr Valialkin
47cb79198e
docs/vmagent.md: clarify how -promscrape.seriesLimitPerTarget command-line flag, series_limit config option and __series_limit__ label interact with each other
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5663
See also 89e3c70ccd
2024-01-23 13:15:45 +02:00
Roman Khavronenko
8461add541
lib/promscrape: respect 0 value for series_limit param (#5663)
* lib/promscrape: respect `0` value for `series_limit` param

Respect `0` value for `series_limit` param in `scrape_config`
even if global limit was set via `-promscrape.seriesLimitPerTarget`.
Previously, `0` value will be ignored in favor of `-promscrape.seriesLimitPerTarget`.

This behavior aligns with possibility to override `series_limit` value via
relabeling with `__series_limit__` label.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 13:09:36 +02:00
Fred Navruzov
7b1325c3a3
- fix 404 errors after page remaning (#5664)
- slight text fixes
2024-01-23 12:59:01 +02:00
Yury Molodov
1db2b991b7
vmui: query report (#5497)
* vmui: add query analyzer page

* vmui: fix tabs for query analyzer

* vmui: add help to export query

* vmui: add time params to query analyzer

* docs/vmui: add query analyzer

* vmui: fix validation JSON form

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 04:26:04 +02:00
Yury Molodov
3a26e4d6ec
vmui: add flag for default timezone setting (#5611)
* vmui: add flag for default timezone setting #5375

* vmui: validate timezone before client return

* Update app/vmselect/vmui.go

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 04:15:14 +02:00
Yury Molodov
574d69775e
vmui: fix cache autocomplete (#5591)
* vmui: fix the logic of closing the popper #5470

* vmui: fix the logic of caching autocomplete results #5472

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 04:06:39 +02:00
Daria Karavaieva
7ad16c8f17
add 1.8.0 notes to changelog (#5616)
* add 1.8.0 notes to changelog

* added release date

* MAD internal link

* monitoring health deprecation
2024-01-23 02:31:21 +02:00
Jaskeerat Singh Randhawa
291802ea84
custom-resources: fix link text for alertmanager (#5660) 2024-01-23 02:30:51 +02:00
Aliaksandr Valialkin
cd4f641d32
app/{vmstorage,vmselect}: disable vmstorage->vmselect RPC compression by default in order to improve query performance 2024-01-23 02:29:13 +02:00
Zakhar Bessarab
60ef978ffc
lib/storage: print tenant ID in log when discarding or truncating labels (#5658)
Previously, it was not possible to determine which tenant sends metrics with excessive amount of labels of label values.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 02:27:59 +02:00
Dmytro Kozlov
69e59ac9b7
deployment/docker: add grafana datasource to the docker-compose files (#5363)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3920
https://github.com/VictoriaMetrics/grafana-datasource/issues/113
2024-01-22 18:43:16 +02:00
hagen1778
ede466be56
docs: fix Grafana link example for vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 18:41:38 +02:00
Aliaksandr Valialkin
c6f6f094c5
Revert "lib/promscrape: do not store last scrape response when stale markers … (#5577)"
This reverts commit cfec258803.

Reason for revert: the original code already doesn't store the last scrape response when stale markers are disabled.
The scrapeWork.areIdenticalSeries() function always returns true is stale markers are disabled.
This prevents from storing the last response at scrapeWork.processScrapedData().

It looks like the reverted commit could also return back the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5577
2024-01-22 01:46:12 +02:00
Aliaksandr Valialkin
3230525c36
docs: use persistent links to Grafana dashboards
These links do not depend on the dashboard name, so they do not break after the renaming of the dashboard.

This is a follow-up for ff33e60a3d
2024-01-22 01:45:42 +02:00
Aliaksandr Valialkin
d4a1a28543
app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery()
This is a follow-up for cf03e11d89

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5630
2024-01-22 01:39:27 +02:00
Artem Navoiev
fbbd5ab1e7
docs vmanomaly fix anchor
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-22 01:36:22 +02:00
Hui Wang
49fa92c1d0
lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice (#5557)
* lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice

Previously the groupWatcher could be mistakenly stopped when requests for pod or services resources take too long.

* remove mislead comment

* docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640

* wip

* lib/promscrape/kubernetes: prevent from stopping groupWatcher when there are in-flight apiWatcher.mustStart() calls

groupWatcher is stopped if it has zero registered apiWatchers during 14 seconds.
But such a groupWatcher can be still in use if apiWatcher for `role: endpoints` or `role: endpointslice`
is being registered and the discovery of the associated `pod` and/or `service` objects takes longer
than 14 seconds - see the beginning of groupWatcher.startWatchersForRole() function for details.

Track the number of in-flight calls to apiWatcher.mustStart() and prevent from stopping the associated groupWatcher
if the number of in-flight calls is non-zero.

P.S. postponing the discovery of `pod` and/or `service` objects associated with `endpoints` or `endpointslice` roles
isn't the best solution, since it slows down initial discovery of `endpoints` and `endpointslice` targets.

* typo fix

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-22 01:33:17 +02:00
Aliaksandr Valialkin
885ee160c2
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-22 01:23:23 +02:00
Fred Navruzov
b76c489e57
- fix 404 links after renaming (#5653)
- improve wording on diagram
- swap enterprise/about chapters for page clarity
2024-01-22 01:14:54 +02:00
Fred Navruzov
11c5b5d3ab
docs: vmanomaly - add component interaction diagram (#5652)
* add interaction diagram for vmanomaly components
* small docs fixes
* resolve suggestions
2024-01-22 01:14:16 +02:00
Aliaksandr Valialkin
6d91d10cbd
docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640
2024-01-22 01:13:19 +02:00
Nikolay
73c51072e6
app/vmauth: adds metric_labels and backend_errors counter (#5585)
* app/vmauth: adds metric_labels and backend_errors counter
it must improve observability for user requests with new metric - per user backend errors counter.
it's needed to calculate requests fail rate to the configured backends.
metric_labels configuration allows to perform additional aggregations on top of multiple users from configuration section.
It could be multiple clients or clients with separate read/write tokens
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-22 01:09:51 +02:00
Aliaksandr Valialkin
2c7c812a9d
lib/promscrape/discovery/kubernetes: add -promscrape.kubernetes.attachNodeMetadataAll command-line flag
This flag allows setting attach_metadata.node=true for all the kubernetes_sd_configs defined at -promscrape.config

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640

Thanks to wasim-nihal for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593
2024-01-22 01:08:52 +02:00
Hui Wang
e086ef16da
app/vmselect/promql: properly handle possible negative results caused… (#5608)
* app/vmselect/promql: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase()

* fix test
2024-01-22 01:04:50 +02:00
Nikolay
d79a7c36b0
app/vmselect/netstorage (#5649)
* app/vmselect/netstorage

correctly handle errGlobal set

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-22 01:01:38 +02:00
Nikolay
e196c61e36
app/vmselect: abort streaming connections for vmselect (#5650)
* app/vmselect: abort streaming connections for vmselect
due to streaming nature of export APIs, curl and simmilr tools cannot
detect errors that happened after http.Header with status 200 was
written to it.

This PR tracks if body write was already started and closes connection.

It allows client to detect not expected chunk sequence and return error
to the caller.

Mostly it affects vmselect at cluster version

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5650

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-22 00:54:32 +02:00
Aliaksandr Valialkin
c05982bfa7
lib/promscrape/discovery/hetzner: follow-up after 03a97dc678
- docs/sd_configs.md: moved hetzner_sd_configs docs to the correct place according to alphabetical order of SD names,
  document missing __meta_hetzner_role label.
- lib/promscrape/config.go: added missing MustStop() call for Hetzner SD,
  and moved the code to the correct place according to alphabetical order of SD names.
- lib/promscrape/discovery/hetzner: properly handle pagination for hloud API responses,
  populate missing __meta_hetzner_role label like Prometheus does.
- Properly populate __meta_hetzner_public_ipv6_network label like Prometheus does.
- Remove unused SDConfig.Token.
- Remove "omitempty" annotation from SDConfig.Role field, since this field is mandatory.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154
2024-01-22 00:53:23 +02:00
Artem Navoiev
1e14c3177b
vmanomly use proper title for overiview doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-22 00:52:56 +02:00