Commit Graph

4565 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
a5445e09c2
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 5ecf439078.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin
813a7aec6c
docs: return back spellcheck fixes from the commit 92b22581e6, which were accidentally reverted in 45cf83cd8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470
2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow (#6470)"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin
7026498359
lib/httpserver: revert 9b7e532172
Reason for revert: this commit doesn't resolve real security issues,
while it complicates the resulting code in subtle ways (aka security circus).

Comparison of two strings (passwords, auth keys) takes a few nanoseconds.
This comparison is performed in non-trivial http handler, which takes thousands
of nanoseconds, and the request handler timing is non-deterministic because of Go runtime,
Go GC and other concurrently executed goroutines. The request handler timing is even
more non-deterministic when the application is executed in shared environments
such as Kubernetes, where many other applications may run on the same host and use
shared resources of this host (CPU, RAM bandwidth, network bandwidth).

Additionally, it is expected that the passwords and auth keys are passed via TLS-encrypted connections.
Establishing TLS connections takes additional non-trivial time (millions of nanoseconds),
which depends on many factors such as network latency, network congestion, etc.

This makes impossible to conduct timing attack on passwords and auth keys in VictoriaMetrics components.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6423/files
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392
2024-06-25 01:51:06 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
8dd69fd7ae
docs/CHANGELOG.md: cut v1.102.0-rc2 release 2024-06-24 23:22:45 +02:00
Nikolay
7ab046cf88
vendor update: updates metrics to v1.34.0 (#6523)
mention change for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1af13208c2)
2024-06-24 14:52:32 +02:00
Ivan Yatskevich
50343768fd
Publish Alertmanager configuration docs for Managed VictoriaMetrics (#6428)
### Describe Your Changes

With the recent release of Managed VictoriaMetrics users are able to
create and execute Alerting & Recording rules and send notifications via
hosted Alertmanager.

So, we're publishing Alertmanager configuration docs for Managed
VictoriaMetrics.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1106d91741)
2024-06-24 12:50:15 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow (#6470)
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Andrii Chubatiuk
50783fca4d
app/vmagent: add max_scrape_size to scrape config (#6434)
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1e83598be3)
2024-06-20 14:00:22 +02:00
Hui Wang
a393b993d6
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512

(cherry picked from commit bc37b279aa)
2024-06-20 14:00:19 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
Zakhar Bessarab
886f545f81
lib/fs/fscore: do not trim content from path (#6503)
### Describe Your Changes

Trimming content which is loaded from an external pass leads to obscure
issues in case user-defined input contained trimmed chars. For example.
user-defined password "foo\n" will become "foo" while user will expect
it to contain a new line.

---
For example, a user defines a password which ends with `\n`. This often
happens when user Kubernetes secrets and manually encodes value as
base64-encoded string.

In this case vmauth configuration might look like:
```
users:
  - url_prefix:
      - http://vminsert:8480/insert/0/prometheus/api/v1/write
    name: foo
    username: foo
    password: "foobar\n"
```

vmagent configuration for this setup will use the following flags:
```
-remoteWrite.url=http://vmauth:8427/
-remoteWrite.basicAuth.passwordFile=/tmp/vmagent-password
-remoteWrite.basicAuth.username="foo"
```
Where `/tmp/vmagent-password` is a file with `foobar\n` password.

Before this change such configuration will result in `401 Unauthorized`
response received by vmagent since after file content will become
`foobar`.

---
An example with Kubernetes operator which uses a secret to reference the
same password in multiple configurations.

<details>
  <summary>See full manifests</summary>

`Secret`:
```
apiVersion: v1
data:
  name: Zm9v # foo
  password: Zm9vYmFy # foobar\n
  username: Zm9v= # foo
kind: Secret
metadata:
  name: vmuser
```

`VMUser`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
  name: vmagents
spec:
  generatePassword: false
  name: vmagents
  targetRefs:
  - crd:
      kind: VMAgent
      name: some-other-agent
      namespace: example
  username: foo
  # note - the secret above is referenced to provide password
  passwordRef:
    name: vmagent
    key: password
```

`VMAgent`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: example
spec:
  selectAllByDefault: true
  scrapeInterval: 5s
  replicaCount: 1
  remoteWrite:
    - url: "http://vmauth-vmauth-example:8427/api/v1/write"
      # note - the secret above is referenced as well
      basicAuth:
        username:
          name: vmagent
          key: username
        password:
          name: vmagent
          key: password
```

</details>

Since both config target exactly the same `Secret` object it is expected
to work, but apparently the result will be `401 Unauthrized` error.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 201fd6de1e)
2024-06-19 10:37:12 +02:00
Nihal
8fd46caa22
victoria-metrics: constant-time comparison of credentials like authkeys and basic auth credentials (#6423)
Changes for constant-time comparison of credentials like authkeys and
basic auth credentials.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392

---------

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
(cherry picked from commit 9b7e532172)
2024-06-19 10:37:09 +02:00
Hui Wang
36f6a08f07
doc: add missing cmd flag description (#6504)
(cherry picked from commit 2259e3cac7)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 15:29:11 +02:00
Yury Molodov
88650abf97
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.

(cherry picked from commit 32fbffedd9)
2024-06-18 15:28:56 +02:00
Hui Wang
5be2f2c4e4
vmalert-tool: support file path with hierarchical patterns and regexp… (#6501)
…es, and http url in unittest cmd-line flag `-files`

(cherry picked from commit 3b8970802e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:17:42 +02:00
Hui Wang
b09da16899
vmalert-tool: exit normally when no rule is defined under rule group (#6502)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit d62f303e53)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:17:23 +02:00
Aliaksandr Valialkin
8650809435
docs/keyConcepts.md: document optional timeout query arg at /api/v1/query and /api/v1/query_range
This is a follow-up for ac9c2a796f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6020
2024-06-18 12:32:04 +02:00
Aliaksandr Valialkin
0ee360b48c
docs/VictoriaLogs/CHANGELOG.md: typo fix in the release date for v0.20.2-victorialogs 2024-06-18 00:28:49 +02:00
Aliaksandr Valialkin
ee114ca59c
app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint
This has been broken in 2b6a634ec0
2024-06-18 00:24:11 +02:00
Aliaksandr Valialkin
7658cc36a6
docs/VictoriaLogs/CHANGELOG.md: cut v0.20.1-victorialogs 2024-06-17 23:29:28 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
f74e6b0674
app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425 2024-06-17 22:30:28 +02:00
jackyin
4a6bf7f218
app/vmui: copy button shows undefined (#6495)
### Describe Your Changes
fix #6421
some aggregation func don't return \_\_name\_\_ value
2024-06-17 22:30:28 +02:00
Roman Khavronenko
df7e300071
app/vmselect/promql: check for ranged vectors in aggr funcs if implicit conversions are disabled (#6450)
Check for ranged vector arguments in aggregate expressions when
`-search.disableImplicitConversion` or `-search.logImplicitConversion`
are enabled.
 For example, `sum(up[5m])` will fail to execute if these flags are set.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [*] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6149adbe10)
2024-06-17 14:25:43 +02:00
Aliaksandr Valialkin
938862946c
docs/VictoriaLogs: mention which collectors support sending logs to VictoriaLogs in syslog format 2024-06-17 13:13:51 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
Zhu Jiekun
9c3dabccac
docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494)
### Describe Your Changes
Add CHANGELOG for Statsd support in v1.102.0-rc1
- CHANGELOG is missing for #5053.
- It would be better to include it so that users can be aware of when
this feature is released, rather than attempting to use it with lower
version VM components. See discussion
[here](https://victoriametrics.slack.com/archives/CGZF1H6L9/p1718462859065049?thread_ts=1718451117.668789&cid=CGZF1H6L9).

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
Fred Navruzov
73fbce30bd
docs/vmanomaly - add config example for min_dev arg (#6490)
### Describe Your Changes

Added config example for `min_dev_from_expected` arg; also, small
styling fixes and alignments

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
hagen1778
43c53271bf
docs: update wording for 6e395048d3
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit da4fbf61a4)
2024-06-14 16:31:42 +02:00
Hui Wang
5ecf439078
app/vmselect: fix the way of counting raw samples in single query (#6464)
The limit is specified with command-line flag
`-search.maxSamplesPerQuery`.
Previously, samples might be over-counted and query can't be fixed by
reducing time range.
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

(cherry picked from commit 6e395048d3)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 16:24:31 +02:00
Andrii Chubatiuk
8ca1813bd2
lib/flagutil: use month limit for duration flag for parsed duration assessment (#6486)
use maxMonths limit for parsed duration flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit faf67aa8b5)
2024-06-14 15:21:32 +02:00
jackyin
f69495cd5f
app/vmalert: fix VMAlert oauth2 error (#6478)
Properly set ClientSecret param for notifier.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5223981fed)
2024-06-14 15:21:30 +02:00
hagen1778
64df1383e7
app/vmalert: properly configure authentication with S3 when -s3.configFilePath is specified.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 14:22:35 +02:00
Andrii Chubatiuk
abc233a902
lib/backup/s3remote: fixed credsFilePath flag (#6488)
properly use credsFilePath flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e678a9aa51)
2024-06-14 14:14:58 +02:00
Zakhar Bessarab
a91a2b97ed
docs/guides/otel-guide: update sentence to make it less obscure (#6483)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit f11261dc44)
2024-06-14 14:14:56 +02:00
Andrii Chubatiuk
779436bd9c
app/vmalert: fixed path prefixes for system routes (#6435)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit eea361defb)
2024-06-14 14:14:54 +02:00
Artem Navoiev
6c587eefbf
docs: add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit d1baa41b6a)
2024-06-14 13:26:47 +02:00
Fred Navruzov
455d761de9
docs/vmanomaly: fix-typos-in-v1.13-preset (#6484)
### Describe Your Changes

small fix of typos in v1.13 presets (vmanomaly docs)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1cf28598c6)
2024-06-14 13:26:44 +02:00
Roman Khavronenko
5df50e5645
lib/streamaggr: prevent rate_sum and rate_avg from producing NaNs (#6482)
### Describe Your Changes

* check if `lastValue` was seen at least twice with different
timestamps. Otherwise, the difference between last timestamp and
previous timestamp could be `0` and will result into `NaN` calculation
* check if there items left in lastValue map after staleness cleanup.
Otherwise, `rate_avg` could have produce `NaN` result.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 51d19485bb)
2024-06-14 13:26:42 +02:00
Github Actions
5683610bf8
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@bc562ef (#6481)
(cherry picked from commit 2d575cd27a)
2024-06-14 13:26:40 +02:00
Fred Navruzov
aa85b772bb
docs/vmanomaly: update preset guide and cross-links (#6480)
### Describe Your Changes

Updates and cross-references for `preset` mode of `vmanomaly` [in
docs](https://docs.victoriametrics.com/anomaly-detection/presets/)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2184c0c730)
2024-06-14 13:26:37 +02:00
Daria Karavaieva
ae39b02496
docs/vmanomaly: preset guide (#6151)
- added instructions of how to run presets
- description of Node Exporter indicators covered

(cherry picked from commit c32b8d39dd)
2024-06-14 13:26:29 +02:00
Aliaksandr Valialkin
b455d356fc
docs/CONTRIBUTING.md: add checks about backwards compatibility and historical conventions 2024-06-13 17:08:54 +02:00
Aliaksandr Valialkin
7af2b7905f
docs/CONTRIBUTING.md: various typo fixes 2024-06-13 17:02:05 +02:00
Zakhar Bessarab
ac16d1dc1b
lib/promscrape: increase default value for promscrape.maxDroppedTargets to 10_000 (#6459)
### Describe Your Changes
This limit can be increased since after
4513893ead
tracking of dropped targets uses much less memory per entry.

See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381#issuecomment-2156708228

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit 34071ac660)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-13 09:28:16 +02:00
LHHDZ
41e4135371
app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401)
This change fixes the following panic:
```
2024-06-04T11:16:52.899Z        warn    app/vmauth/auth_config.go:353   cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally
panic: runtime error: integer divide by zero

goroutine 9 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58
panic({0x103115100?, 0x10338d700?})
        /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124
main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210
main.(*URLPrefix).getBackendURL(0x140000aa080)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8
```

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-06-12 11:47:44 +02:00
Aliaksandr Valialkin
27efb9499b
docs/VictoriaLogs/logsql-examples.md: added an example on how to search for a phrase with whitespace 2024-06-11 19:13:47 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress 2024-06-11 17:51:01 +02:00
Fred Navruzov
ea90d91a30
docs/vmanomaly - fix 404 img links & remove outdated models (#6460)
### Describe Your Changes

Fix 404 relative img links for v1.13.0 update of vmanomaly docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 9a355c3512)
2024-06-11 17:05:10 +02:00
Fred Navruzov
0219d34b21
docs/vmanomaly - release 1.13.0 preparation (#6436)
### Describe Your Changes

[vmanomaly docs](https://docs.victoriametrics.com/anomaly-detection/)
update for changes, introduced in v1.13.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1feb5d04d7)
2024-06-11 17:05:07 +02:00
Nikolay
ae76794a19
follow-up docs update after 77f22fdb8d (#6454)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 33d07e915f)
2024-06-11 17:05:05 +02:00
Github Actions
c96b334b3e
Automatic update operator docs from VictoriaMetrics/operator@310a860 (#6449)
(cherry picked from commit 020d3f3d88)
2024-06-11 17:05:01 +02:00
Lapo Luchini
dd311aa76f
Small fix in MetricsQL.md (#6452)
### Describe Your Changes

value→valid

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Lapo Luchini <lapo@lapo.it>
(cherry picked from commit 14f586238f)
2024-06-11 17:04:59 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
Yury Molodov
2300e30ff3
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:39:25 +02:00
hagen1778
fdf0a936f0
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8d95522529)
2024-06-10 14:05:58 +02:00
Nikolay
0ce7f38e1c
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d44058bcd6)
2024-06-10 12:41:51 +02:00
Dmytro Kozlov
a4bdc14bc5
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 8b46bb0c41)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:44 +02:00
Hui Wang
028a80613f
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329,
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 61dce6f2a1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:29 +02:00
Andrii Chubatiuk
6fd314d8ba
vmagent: updated dashboard and alert for stream aggregation (#6427)
### Describe Your Changes

Added streaming aggregation section to vmagent dashboards
Added alert for streaming aggregation and deduplication flush timeouts
Removed deprecated compose versions from compose files

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2da45a8368)
2024-06-10 12:37:22 +02:00
Artem Navoiev
d6b56a1460
docs victorialogs: add alias for quickstart for VL to be consistent with other quickstarts
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 318e9e9de0)
2024-06-10 12:37:19 +02:00
Artem Navoiev
a383e00e48
add alias for vmalert-tools
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 8f254232bb)
2024-06-10 12:37:16 +02:00
Aliaksandr Valialkin
792dd697f4
docs/CHANGELOG.md: document v1.93.15 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.15
2024-06-07 23:43:20 +02:00
Aliaksandr Valialkin
f61bf790a4
docs/LTS-releases.md: update v1.97.4 LTS release to v1.97.5
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:16:55 +02:00
Aliaksandr Valialkin
37ca3a2b0c
docs/CHANGELOG.md: add changelog for v1.97.5 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:03:35 +02:00
Aliaksandr Valialkin
3f883559e2
docs/CHANGELOG.md: cut v1.202.0-rc1 release 2024-06-07 16:53:48 +02:00
Aliaksandr Valialkin
a38759f66a
docs: run make docs-sync after c740a8042e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362
2024-06-07 16:45:53 +02:00
Aliaksandr Valialkin
32aa0751a1
lib/streamaggr: follow-up for 7cb894a777
- Use bytesutil.InternString() instead of strings.Clone() for inputKey and outputKey in aggregatorpushSamples().
  This should reduce string allocation rate, since strings can be re-used between aggrState flushes.
- Reduce memory allocations at dedupAggrShard by storing dedupAggrSample by value in the active series map.
- Remove duplicate call to bytesutil.InternBytes() at Deduplicator, since it is already called inside dedupAggr.pushSamples().
- Add missing string interning at rateAggrState.pushSamples().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402
2024-06-07 16:35:53 +02:00
Roman Khavronenko
78121642df
lib/streamaggr: reduce number of inuse objects (#6402)
The main change is getting rid of interning of sample key. It was
discovered that for cases with many unique time series aggregated by
vmagent interned keys could grow up to hundreds of millions of objects.
This has negative impact on the following aspects:
1. It slows down garbage collection cycles, as GC has to scan all inuse
objects periodically. The higher is the number of inuse objects, the
longer it takes/the more CPU it takes.
2. It slows down the hot path of samples aggregation where each key
needs to be looked up in the map first.

The change makes code more fragile, but suppose to provide performance
optimization for heavy-loaded vmagents with stream aggregation enabled.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-06-07 16:35:52 +02:00
Roman Khavronenko
fae589bb83
lib/promrelabel: speedup label match by __name__ (#6432)
The change adds a fastpath for `equalValue` comparisons against
`__name__` label by avoiding calls to `toCanonicalLabelName` func. This
speedups matches by metric name like `'foo'`. See bench stats below:
```
benchcmp old.txt new.txt

benchmark                                           old ns/op     new ns/op     delta
BenchmarkIfExpression/equal_label:_last-10          35.6          35.1          -1.18%
BenchmarkIfExpression/equal_label:_middle-10        18.3          17.3          -5.41%
BenchmarkIfExpression/equal_label:_first-10         1.20          1.24          +2.74%
BenchmarkIfExpression/equal___name__:_last-10       10.1          4.96          -50.75%
BenchmarkIfExpression/equal___name__:_middle-10     5.79          3.16          -45.41%
BenchmarkIfExpression/equal___name__:_first-10      1.17          1.05          -9.76%
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-07 16:35:52 +02:00
Andrii Chubatiuk
93cd08f15f
lib/streamaggr: metrics to track dropped, nan samples and samples lag (#6358)
### Describe Your Changes

Added streamaggr metrics to:
 - `vm_streamaggr_samples_lag_seconds` - samples lag
- `vm_streamaggr_ignored_samples_total{reason="nan"}` - ignored NaN
samples
- `vm_streamaggr_ignored_samples_total{reason="too_old"}` - ignored old
samples

(cherry picked from commit 185fac03b3)
2024-06-06 19:22:45 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress 2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
6d986f731b
docs/VictoriaLogs/CHANGELOG.md: document the a68c2c0f17 2024-06-06 12:21:47 +02:00
Github Actions
2e3c039113
Automatic update operator docs from VictoriaMetrics/operator@32f7674 (#6414) 2024-06-06 12:12:58 +02:00
Aliaksandr Valialkin
c6b6900152
docs/VictoriaLogs/LogsQL.md: typo fix: replace from _json with from _msg where appropriate 2024-06-05 10:15:56 +02:00
Aliaksandr Valialkin
be4aefd133
docs/VictoriaLogs/LogsQL.md: substitute TAG with APP-NAME[PROCID] in rfc3164 syslog message format at unpack_syslog pipe docs 2024-06-05 10:13:58 +02:00
Aliaksandr Valialkin
a200fb433a
lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin
f8336783c8
docs/VictoriaLogs/LogsQL.md: typo fix: substitute "'" with "" in front of John` 2024-06-05 09:51:29 +02:00
Aliaksandr Valialkin
9e79a632c1
docs/VictoriaLogs/LogsQL.md: typo fix: its -> it 2024-06-05 09:46:35 +02:00
Zhu Jiekun
024ab786ce
docs: [deployment] update CHANGELOG.md to include go version change 1.22.4 (#6412)
### Describe Your Changes
Update CHANGELOG to include go version change introduced in
43cf221681

Also see:
- https://go.dev/doc/devel/release
-
https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-05 09:46:34 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
hagen1778
607007add9
docs: add technical articles by @jiekun
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-05 03:16:41 +02:00
pludov
2efd97a63c
lib/fs: support NFS implementations that return EEXIST instead of ENOTEMPTY (#6398)
### Describe Your Changes

Fix for issue #6396: according to rmdir manpage, ENOTEMPTY and EEXIST
should be treated equally

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Ludovic Pollet <ludovic.pollet@exfo.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3ddae77c63)
2024-06-04 15:30:48 +02:00
Aliaksandr Valialkin
1a1ee87b46
docs/VictoriaLogs/LogsQL.md: mention unpack_syslog pipe in the Transformations section 2024-06-04 03:17:14 +02:00
Aliaksandr Valialkin
7d31574071
docs/VictoriaLogs/Roadmap.md: remove LogsQL transformations from roadmap, since they are already implemented 2024-06-04 02:21:23 +02:00
Aliaksandr Valialkin
39d238592c
docs/VictoriaLogs/logsql-examples.md: typo fix 2024-06-04 02:17:59 +02:00
Aliaksandr Valialkin
53928a44d1
docs/VictoriaLogs/logsql-examples.md: typo fix 2024-06-04 02:10:56 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
hagen1778
d6096a477f
app/vmalert: rm extra response for unsupported path
Unsupported path is already handled by `lib/httpserver`.
This prevents from misleading errors in logs caused by double-writing response headers.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a5f81f67fd)
2024-06-03 12:53:38 +02:00
Hui Wang
39286afae3
vmalert-tool: fix float values template in input_series (#6395)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6391

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit e3e40cb848)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:53:58 +02:00
Zakhar Bessarab
ff1bf76237
deployment/docker: add scratch-based images (#6386)
### Describe Your Changes

Scratch based images will be using a separate tag: "(version)-scratch"
and will be built for the same architecture as regular images.
This is useful for environments with higher security standards. In this
case using alpine as base layer requires updating images more frequently
in order to get the latest updates for the base image, even in case the
user did not need to update VictoriaMetrics version.

Tested that scratch images work for:
- vmagent - enterprise with kafka and opensource
- cluster
- single-node

No issues observed so far.

cc: @tenmozes

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7dc9124ba7)
2024-06-03 11:53:45 +02:00
Github Actions
32a9c62481
Automatic update operator docs from VictoriaMetrics/operator@d936bb7 (#6378)
(cherry picked from commit 53422797a7)
2024-06-03 11:53:39 +02:00
hagen1778
73c9981335
chore: follow-up after c740a8042e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6d8e02f278)
2024-06-03 11:53:37 +02:00
Arkadii Yakovets
a6655322b1
docs: fix docs/ and README.md spelling errors (#6362)
Fixes `docs/` and `README.md` typos and errors.

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>

(cherry picked from commit c740a8042e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:53:33 +02:00
Nikolay
908a50f79d
app/vmalert: adds idleConnTimeout flags and retry trivial network errors (#6382)
* "*.idleConnTimeout" flags must reduce probability of `write: broken
pipe` and `read: connection reset by peer` errors Those errors may occur
if remote server closes TCP socket for connection, while it's still
exist at client.
* single time retries for `write: broken pipe` and `read: connection
reset by peer` must handle a case for incorrectly configured timeouts at
middleware proxies, mitigate minor network issues.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit b97916276f)
2024-06-03 11:52:58 +02:00
Aliaksandr Valialkin
b994e64567
docs/VictoriaLogs: typo fixes 2024-05-30 16:35:25 +02:00
Aliaksandr Valialkin
2fd8cf15c4
deployment/docker: update VictoriaLogs Docker image from v0.14.0-victorialogs to v0.15.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.15.0-victorialogs
2024-05-30 16:27:53 +02:00
Aliaksandr Valialkin
540bbb63a2
lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
Zakhar Bessarab
b08ee34061
docs/metricsql: fix typo in link (#6384)
### Describe Your Changes

Fix typo in link.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-05-30 16:17:20 +02:00
Fred Navruzov
b200c4263a
docs/vmanomaly: improve FAQ (#6369)
### Describe Your Change
More explicit [vmanomaly
FAQ](https://docs.victoriametrics.com/anomaly-detection/faq/index.html),
based on common Q&A from recent communications with users


### Checklist

The following checks are **mandatory**:

- ✔️ My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-05-30 16:16:56 +02:00
Dmytro Kozlov
617ec1fbec
docs: update managed create deployment description (#6376)
### Describe Your Changes

Updated description how to create deployment in the cloud soulution
2024-05-30 16:16:18 +02:00
yumeiyin
95b8cf76f8
chore: remove redundant words (#6348)
(cherry picked from commit 9289c7512d)
2024-05-29 14:37:04 +02:00
Roman Khavronenko
189af53142
lib/storage: filter deleted label names and values from `/api/v1/labe… (#6342)
…ls` and `/api/v1/label/.../values`

Check for deleted metrics when `match[]` filter matches small number of
time series (optimized path).

The issue was introduced
[v1.81.0](https://docs.victoriametrics.com/changelog_2022/#v1810).

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6300 Updates
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit b984f4672e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-29 14:37:00 +02:00
Github Actions
9d301809b6
Automatic update operator docs from VictoriaMetrics/operator@cf753a6 (#6373)
(cherry picked from commit 14ba7b237d)
2024-05-29 14:04:24 +02:00
Aliaksandr Valialkin
e83fd4a117
lib/logstorage: work-in-progress 2024-05-29 01:52:34 +02:00
Aliaksandr Valialkin
6ddc618731
docs/VictoriaLogs: typo fixes 2024-05-28 22:48:57 +02:00
Aliaksandr Valialkin
2395c7e65e
deployment/docker: update VictoriaLogs image from v0.12.1-victorialogs to v0.13.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.13.0-victorialogs
2024-05-28 22:16:30 +02:00
Alexander Marshalov
7d532a31fb
Update base Alpine image to 3.20.0 to avoid security risks (#6370)
fixes: CVE-2023-42366, CVE-2023-42363, CVE-2024-4603, CVE-2024-2511,
CVE-2024-24788, CVE-2024-24787
2024-05-28 22:16:29 +02:00
Aliaksandr Valialkin
79c03fc35f
lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin
d429c0b83e
deployment: update VictoriaLogs image from v0.12.0-victorialogs to v0.12.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.1-victorialogs
2024-05-26 02:08:52 +02:00
Aliaksandr Valialkin
afa597ce2a
lib/logstorage: work-in-progress 2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin
6427b3c3c0
lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
Aliaksandr Valialkin
f02c5cb79a
docs/VictoriaLogs/CHANGELOG.md: cut v0.12.0-victorialogs 2024-05-25 21:40:20 +02:00
Aliaksandr Valialkin
03fe4c8963
lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress 2024-05-25 00:31:55 +02:00
hagen1778
63b83d62e8
deployment/alerts: add new alerting rules TooLongLabelValues and TooLongLabelNames to notify about truncation of label values or names respectively.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1be1e9a7a4)
2024-05-24 16:08:41 +02:00
Nikolay
5025ede7bc
lib/mergeset: adds tracking for indexdb records drop (#6297)
It allows to create alert for possible item drops at indexdb. It may
happen, if ingested metric size exceeds max indexdb item size.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 69d244e6fb)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-24 16:08:34 +02:00
Zakhar Bessarab
85eee7de0a
app/vmselect: update flag description (#6347)
Update wording to highlight that cache is not persistent if flag is
value is empty. Previously, it was not clear if cache is not used at all
or just not persistent.
2024-05-24 15:58:54 +02:00
Aliaksandr Valialkin
db48642302
deployment: update Docker image tag for VictoriaLogs from v0.8.0-victorialogs to v0.10.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.10.0-victorialogs
2024-05-24 03:16:06 +02:00
Aliaksandr Valialkin
e5ca4fff76
docs/VictoriaLogs/CHANGELOG.md: cut v0.10.0-victorialogs 2024-05-24 03:08:29 +02:00
Aliaksandr Valialkin
7a2a2f173e
lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
Github Actions
8025cfdb73
Automatic update operator docs from VictoriaMetrics/operator@4b059a4 (#6332) 2024-05-24 03:05:39 +02:00
hagen1778
d691d515f1
docs: fix changelog formatting
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-24 03:05:39 +02:00
Aliaksandr Valialkin
ae8b2bcf2e
app/vlselect: fix loading web UI 2024-05-22 23:25:00 +02:00
Aliaksandr Valialkin
2befafa264
docs/VictoriaLogs/LogsQL.md: mention that it is better from performance PoV parsing JSON and logfmt logs before the ingestion into VictoriaLogs 2024-05-22 23:04:48 +02:00
Aliaksandr Valialkin
181b5c5437
docs/VictoriaLogs/LogsQL.md: typo fixes 2024-05-22 22:24:56 +02:00
Aliaksandr Valialkin
802d7edd0e
docs/VictoriaLogs/CHANGELOG.md: cut v0.8.0-victorialogs 2024-05-22 22:07:38 +02:00
Nikolay
dfbd2f8ff7
lib/storage: change default value for maxLabelValueLen to 1024 (#6313)
* It must reduce memory usage for misbehaving clients. Since
VictoriaMetrics stores sparse index inmemory.
* Reduce disk space usage for indexdb.
* Prevent possible indexDB items drops.
* It may trigger slow insert and new timeseries registration due to
default value for flag change

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:55:21 +02:00
Alexander Marshalov
0b70c4c1f1
[vmlogs] fixed time parsing with millisecond precision time (#6293) (#6295)
fix for #6293

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:54:50 +02:00
Yury Molodov
252a196405
vmui/logs: fix parsing long _msg values (#6310)
This PR fixes an issue where parsing long `_msg` values caused errors,
resulting in some log records not being displayed.

The error occurred due to partial processing of strings. In some cases,
a long record could be split into multiple chunks, causing only part of
the record to be processed instead of the entire entry.

#6281

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:44:49 +02:00
Aliaksandr Valialkin
04d0dd2542
lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
hagen1778
89819f2054
dashboards: use $__interval variable for offsets and look-behind windows in annotations
This should improve precision of `restarts` and `version change` annotations when
 zooming-in/zooming-out on the dashboards.

 The change also makes `restarts` dashboard visible on the panels, so user can disable it from
 displaying if needed. This could be useful when restarts overlap with version change events.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9dd9b4442f)
2024-05-22 16:40:08 +02:00
Roman Khavronenko
f3e893f699
lib/backup: add -s3TLSInsecureSkipVerify command-line flag (#6318)
* The new flag can be used for for skipping TLS certificates
verification when connecting to S3 endpoint. Affects vmbackup,
vmrestore, vmbackupmanager.

* replace deprecated `EndpointResolver` with `BaseEndpoint`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1056

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ac836bcf6c)
2024-05-22 16:40:06 +02:00
Daria Karavaieva
aea042bc9d
docs/vmanomaly: backtest parameter fix (#6327)
### Fix backtest parameters

Fix date parameters names in vmanomaly backtest scheduler component
documentation.

(cherry picked from commit c40f355496)
2024-05-22 16:40:01 +02:00
Hui Wang
5b8c3fc9d0
app/vmalert: support DNS SRV record in -remoteWrite.url (#6299)
part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053,
supports [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) address in
`-remoteWrite.url` command-line option.

(cherry picked from commit d7b5062917)
2024-05-22 10:53:22 +02:00
Hui Wang
0b9e59e67e
vmalert: speed up reloading rules from object storage by verifying ob… (#755)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6210
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 974b7783ee)
2024-05-21 16:37:27 +02:00
Github Actions
26306fec80
Automatic update operator docs from VictoriaMetrics/operator@84833cc (#6316)
(cherry picked from commit 23619a3adf)
2024-05-21 16:37:25 +02:00
hagen1778
0dd3fec2b7
deployment/dashboards: fix AnnotationQueryRunner error in Grafana
The error appears when executing annotations query against Prometheus backend
because the query itself hasn't specified look-behind window (which is allowed
in VictoriaMetrics query engine).

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6309
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c746ba154d)
2024-05-21 16:37:23 +02:00
Artem Navoiev
f4743dcef2
docs: victorialogs mention that our bench suite can be run against Loki as well
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit cee4bfebd7)
2024-05-21 16:37:21 +02:00
Yury Molodov
33eaa18c14
vmui: fix URL params handling for navigation (#6284)
This PR fixes the handling of URL parameters to ensure correct browser
navigation using the back and forward buttons.

#6126

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232
(cherry picked from commit f14497f1cd)
2024-05-20 14:46:41 +02:00
Yury Molodov
97c3c946a7
vmui/logs: change time range to start and end query args (#6296)
change time range limitation from `_time` in the expression to `start`
and `end` query args.

(cherry picked from commit a6a599cbdc)
2024-05-20 14:46:39 +02:00
Roman Khavronenko
3e8b5e74d5
lib/streamaggr: skip empty aggregators (#6307)
Prevent excessive resource usage when stream aggregation config file
contains no matchers by prevent pushing data into Aggregators object.
Before this change a lot of extra work was invoked without reason.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 7ce052b32d)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-20 14:46:36 +02:00
Github Actions
d86d768aec
Automatic update operator docs from VictoriaMetrics/operator@c8534a5 (#6305) 2024-05-20 11:04:36 +02:00
Aliaksandr Valialkin
10dd940c54
docs/VictoriaLogs: typo fixes 2024-05-20 04:48:42 +02:00
Aliaksandr Valialkin
18982e1ab7
docs/VictoriaLogs/LogsQL.md: typo fix 2024-05-20 04:27:44 +02:00
Aliaksandr Valialkin
b02d085a52
docs/VictoriaLogs/LogsQL.md: typo fix 2024-05-20 04:24:34 +02:00
Aliaksandr Valialkin
400f43dd48
deployment: update VictoriaLogs from v0.6.0-victorialogs to v0.8.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.8.0-victorialogs
2024-05-20 04:17:51 +02:00
Aliaksandr Valialkin
582e7d5439
lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
viperstars
ab78f3c89d
app/vmagent/remotewrite: skip sending empty block to downstream server (#6241)
Occasionally, vmagent sends empty blocks to downstream servers. If a
downstream server returns an unexpected response, vmagent gets stuck in
a retry loop. While vmagent handles 400 and 409 errors, there are
various prometheus remote write implementations that return different
error codes. For example, vector returns a 422 error. To mitigate the
risk of vmagent getting stuck in a retry loop, it is advisable to skip
sending empty blocks to downstream servers.

Co-authored-by: hao.peng <hao.peng@smartx.com>
Co-authored-by: Zhu Jiekun <jiekun.dev@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3661373cc2)
2024-05-17 14:57:07 +02:00
Andrii Chubatiuk
23c7193ca4
docs: added information about difference between default_rollup and last_over_time (#6282)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6279
(cherry picked from commit ce3cfd720e)
2024-05-17 14:57:04 +02:00
Yury Molodov
5bfbfe6ad2
vmui: remove redundant requests on the Explore Cardinality page (#6263)
Remove redundant requests on the Explore Cardinality page.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240
(cherry picked from commit be291c36f7)
2024-05-17 14:56:55 +02:00
Yury Molodov
0edef9105b
vmui: fix calendar display (#6255)
Fix the calendar display issue occurring with the `UTC+00:00` timezone

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239
(cherry picked from commit 4ad577cc6f)
2024-05-17 14:56:53 +02:00
Andrii Chubatiuk
fe332c3419
app/vmagent: add global aggregator (#6268)
Add global stream aggregation for VMAgent

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467
(cherry picked from commit f153f54d11)
2024-05-17 14:01:31 +02:00
Nikolay
ee4a94a371
follow-up for c6c5a5a186 (#6265)
* adds datadog extensions for statsd:
  - multiple packed values (v1.1)
  - additional types distribution, histogram

* adds type check and append metric type to the labels with special tag
name `__statsd_metric_type__`. It simplifies streaming aggregation
config.

* remove statsd support from cluster, since cluster doesn't support
stream aggregation.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit b2765c45d0)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-17 13:49:24 +02:00
Roman Khavronenko
a5c427bac4
app/vmalert/datasource: reduce number of allocations when parsing instant responses (#6272)
Allocations are reduced by implementing custom json parser via fastjson
lib.
The change also re-uses `promInstant` object in attempt to reduce number
of
allocations when parsing big responses, as usually happens with heavy
recording rules.

```
name                                old allocs/op  new allocs/op  delta
ParsePrometheusResponse/Instant-10     9.65k ± 0%     5.60k ± 0%   ~     (p=1.000 n=1+1)

```

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4f0525852f)
2024-05-16 09:35:58 +02:00
Aliaksandr Valialkin
473c7a344c
docs/VictoriaLogs/CHANGELOG.md: cut v0.7.0-victorialogs
(cherry picked from commit 6fdba8599d)
2024-05-16 09:35:57 +02:00
Aliaksandr Valialkin
28626db066
lib/logstorage: work-in-progress
(cherry picked from commit 0aa19a2837)
2024-05-16 09:35:55 +02:00
Nikolay
2b2fdffd77
app/vmauth: explicitly unregister metrics set for auth config (#6252)
it's needed to remove Summary metric type from the global state of
metrics package. metrics package tracks each bucket of summary and
periodically swaps old buckets with new.

Simple set unregister is not enough to release memory used by Set

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247
(cherry picked from commit 6a6e34ab8e)
2024-05-14 09:28:37 +02:00
Aliaksandr Valialkin
8af6c80bf2
docs/VictoriaLogs/CHANGELOG.md: cut v0.6.1-victorialogs 2024-05-14 03:07:02 +02:00
Aliaksandr Valialkin
b1ee7bca1a
lib/logstorage: work-in-progress 2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin
f52275bbd7
lib/logstorage: work-in-progress
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258
2024-05-14 01:49:58 +02:00
Hui Wang
ec56f4625e
storage: correctly apply -inmemoryDataFlushInterval when it's set t… (#6221)
…o minimum supported value 1s
pendingRowsFlushInterval was bumped to 2s in
73f0a805e2

(cherry picked from commit 4c80b17027)
2024-05-13 16:50:02 +02:00
Andrii Chubatiuk
d9cddf1ad8
lib/streamaggr: added rate and rate_avg output (#6243)
Added `rate` and `rate_avg` output
Resource usage is the same as for increase output, tested on a benchmark

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9c3d44c8c9)
2024-05-13 16:49:39 +02:00
Andrii Chubatiuk
ec2273b247
app/vmagent: removed deprecated -remoteWrite.multitenantURL flag support (#6253)
Removed deprecated `-remoteWrite.multitenantURL` flag to simplify global
stream aggregation

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 680b8c25c8)
2024-05-13 16:49:33 +02:00
Aliaksandr Valialkin
3c645c4a79
deployment: update VictoriaLogs Docker image from v0.5.2-victorialogs to v0.6.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.0-victorialogs
2024-05-12 23:23:05 +02:00
Aliaksandr Valialkin
b24daf7712
docs/VictoriaLogs/LogsQL.md: cross-reference uniq pipe with uniq_values stats function 2024-05-12 16:48:12 +02:00
Aliaksandr Valialkin
3ba65c5804
docs/VictoriaLogs/CHANGELOG.md: cut v0.6.0-victorialogs 2024-05-12 16:38:09 +02:00
Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin
0d55ef3658
docs/vmauth.md: small fixes after proofreading 2024-05-12 12:35:32 +02:00
Aliaksandr Valialkin
536d87cd51
app/vmselect/promql: properly estimate the needed amounts of memory for executing aggregate function over rollup function in incremental mode
Incremental aggregation processes only GOMAXPROCS time series at a time, so its' memory usage doesn't depend
on the number of input time series.

The issue has been introduced in 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2024-05-12 10:14:27 +02:00
Aliaksandr Valialkin
65128ea814
README.md: mention that -tlsCertFile and -tlsKeyFile options aren't needed when automatic issuing of tls certificates is enabled with -tlsAutoCertHosts flag 2024-05-12 09:49:16 +02:00
Aliaksandr Valialkin
0c89adf589
docs/Single-server-VictoriaMetrics.md and docs/README.md: sync with README.md with make docs-sync after the commit c6c5a5a186 2024-05-12 09:41:39 +02:00
Roman Khavronenko
0bed453737
Feature allow configuring disableOnDiskQueue and dropSamplesOnOverload per url (#6248)
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html):
allow configuring `-remoteWrite.disableOnDiskQueue` and
`-remoteWrite.dropSamplesOnOverload` cmd-line flags per each
`-remoteWrite.url`. See this [pull
request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065).
Thanks to @rbizos for implementaion!
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add
labels `path` and `url` to metrics
`vmagent_remotewrite_push_failures_total` and
`vmagent_remotewrite_samples_dropped_total`. Now number of failed pushes
and dropped samples can be tracked per `-remoteWrite.url`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Raphael Bizos <r.bizos@criteo.com>
(cherry picked from commit 87fd400dfc)
2024-05-10 14:32:23 +02:00
Github Actions
ab5b233c3a
Automatic update operator docs from VictoriaMetrics/operator@0829591 (#6250)
(cherry picked from commit c87ce86d96)
2024-05-10 14:32:18 +02:00
Github Actions
ac33ef71ca
Automatic update operator docs from VictoriaMetrics/operator@4a51b37 (#6245)
(cherry picked from commit a8d0c1a62d)
2024-05-10 14:29:22 +02:00
Roman Khavronenko
7be6fcd8fd
lib/streamaggr: set correct suffix <output>_prometheus (#6228)
Set correct suffix `<output>_prometheus` for aggregation outputs
`increase_prometheus` and `total_prometheus`
Before, outputs `total` and `total_prometheus` or `increase` and
`increase_prometheus` had the same suffix.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8a03e987cb)
2024-05-10 14:29:01 +02:00
Andrii Chubatiuk
f3d65ba902
streamaggr: made labels compressor shared (#6173)
Though labels compressor is quite resource intensive, each aggregator
and deduplicator instance has it's own compressor. Made it shared across
all aggregators to consume less resources while using multiple
aggregators.

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit a9283e06a3)
2024-05-10 14:28:59 +02:00
Zhu Jiekun
139f909cdb
chore: [deployment] upgrade from go 1.22.2 to 1.22.3 to include security fixes (#6238)
### Describe Your Changes

upgrade from go 1.22.2 to 1.22.3 to include security fixes. Also see:
- https://go.dev/doc/devel/release
-
https://github.com/golang/go/issues?q=milestone%3AGo1.22.3+label%3ACherryPickApproved

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Jiekun <jiekun.dev@gmail.com>
(cherry picked from commit 02851d7800)
2024-05-10 14:28:56 +02:00
Zhu Jiekun
10063e98a6
feature: [vmagent] Add service discovery support for Vultr (#6068)
### Describe Your Changes
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041

#### Added
- Added service discovery support for Vultr.

#### Docs
- `CHANGELOG.md`, `sd_configs.md`, `vmagent.md` are updated.

#### Note
- Useful links:
- Vultr API:
https://www.vultr.com/api/#tag/instances/operation/list-instances
    - Vultr client SDK: https://github.com/vultr/govultr
- Prometheus SD:
https://github.com/prometheus/prometheus/tree/main/discovery/vultr

---
### Checklist

The following checks are mandatory:

- [X] I have read the [Contributing
Guidelines](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md)
- [x] All commits are signed and include `Signed-off-by` line. Use `git
commit -s` to include `Signed-off-by` your commits. See this
[doc](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) about
how to sign your commits.
- [x] Tests are passing locally. Use `make test` to run all tests
locally.
- [x] Linting is passing locally. Use `make check-all` to run all
linters locally.

Further checks are optional for External Contributions:

- [X] Include a link to the GitHub issue in the commit message, if issue
exists.
- [x] Mention the change in the
[Changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md).
Explain what has changed and why. If there is a related issue or
documentation change - link them as well.

  Tips for writing a good changelog message::

* Write a human-readable changelog message that describes the problem
and solution.
* Include a link to the issue or pull request in your changelog message.
* Use specific language identifying the fix, such as an error message,
metric name, or flag name.
* Provide a link to the relevant documentation for any new features you
add or modify.

- [ ] After your pull request is merged, please add a message to the
issue with instructions for how to test the fix or try the feature you
added. Here is an
[example](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048#issuecomment-1546453726)
- [x] Do not close the original issue before the change is released.
Please note, in some cases Github can automatically close the issue once
PR is merged. Re-open the issue in such case.
- [x] If the change somehow affects public interfaces (a new flag was
added or updated, or some behavior has changed) - add the corresponding
change to documentation.

Signed-off-by: Jiekun <jiekun.dev@gmail.com>
(cherry picked from commit 17e3d019d2)
2024-05-10 14:28:51 +02:00
Oleg
76af930e4a
Statsd protocol compatibility (#5053)
In this PR I added compatibility with [statsd
protocol](https://github.com/b/statsd_spec) with tags to be able to send
metrics directly from statsd clients to vmagent or directly to VM.
For example its compatible with
[statsd-instrument](https://github.com/Shopify/statsd-instrument) and
[dogstatsd-ruby](https://github.com/DataDog/dogstatsd-ruby) gems

Related issues: #5052, #206, #4600

(cherry picked from commit c6c5a5a186)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-10 14:27:31 +02:00
Github Actions
cc68484f95
Automatic update operator docs from VictoriaMetrics/operator@2789953 (#6237)
(cherry picked from commit 55c7dafb35)
2024-05-10 13:14:24 +02:00
Alexander Marshalov
2922b2a2e0
fix typo in scrape config examples (#6234)
(cherry picked from commit 3d4988ecf6)
2024-05-10 13:14:22 +02:00
Github Actions
bb667fd3f7
Automatic update operator docs from VictoriaMetrics/operator@6271553 (#6233)
(cherry picked from commit 134dcaef33)
2024-05-10 13:14:19 +02:00
hagen1778
33db552a39
docs: mention influxListenAddr in URLs format doc
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2561a132ee)
2024-05-10 13:14:09 +02:00
Andrii Chubatiuk
e26b55db1e
app/vmagent/remotewrite: do not cleanup timeseries which are used in multiple remote write contexts (#6206)
When at least one remote write has deduplication configured it cleans up
timeseries while they can be in use by another remote write without
deduplication

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205
---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 879771808b)
2024-05-06 12:10:45 +02:00
Yury Molodov
75af52c1d0
vmui: fix issue preventing first query trace expansion (#6197)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186
(cherry picked from commit 046a4a5ecf)
2024-04-30 18:39:22 +02:00
hagen1778
798a446d5a
docs: mention actual version in update nodes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c0050beadc)
2024-04-30 18:38:20 +02:00
Hui Wang
abd29c15ab
docs: update vmalert and vmagent docs (#6207)
* restore and actualize doc section explaining duplicated labels error
* rm misleading comment about post-aggregation in stream aggregation

(cherry picked from commit e3c226cf92)
2024-04-30 10:30:19 +02:00
Github Actions
f22b2a0563
Automatic update operator docs from VictoriaMetrics/operator@8f025b3 (#6200)
Co-authored-by: Hui Wang <haley@victoriametrics.com>
(cherry picked from commit e908effd22)
2024-04-30 10:30:10 +02:00
hagen1778
d87c8757cf
dashboards: add new panel Concurrent selects to vmstorage row
The panel will show how many ongoing select queries are processed by vmstorage
and should help to identify resource bottlenecks. See panel description for more details.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d386a68b59)
2024-04-30 10:30:08 +02:00
hagen1778
0d77a55961
deployment: update per-tenant-statistic dashboard to be compatible with Grafana 10
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9e18724036)
2024-04-30 10:30:06 +02:00
hagen1778
a0698d92c3
deployment: update backupmanager dashboard to be compatible with Grafana 10
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5917ac003e)
2024-04-30 10:30:04 +02:00
hagen1778
40af0fa179
deployment: update operator dashboard to be compatible with Grafana 10
- Use TimeSeries panel instead of deprecated Graph
- Update panel styles
- Fix version panel

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f0c4d372bd)
2024-04-30 10:30:02 +02:00
hagen1778
8f48747802
dashboards: add Network Usage panel to Resource Usage row
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4478
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8606b48ce5)
2024-04-30 10:29:58 +02:00
Dima Lazerka
d821e13c24
deployment/dashboards: properly show version for non-stable docker images (#6150)
re: .*-(?:tags|heads)-(.*)-(?:0|dirty)-.*

cases:
victoria-metrics-20240419-160209-heads-enterprise-single-node-0-g08f933ab0c
enterprise-single-node

victoria-metrics-20240201-133950-tags-v1.97.1-enterprise-0-g760a8733b
v1.97.1-enterprise

victoria-metrics-20240419-160209-heads-rotation-part-2-0-ge2367b6d1-dirty-848b54cd
rotation-part-2-0-ge2367b6d1

victoria-metrics-20240419-160209-heads-lts-1.93-enterprise-search-contention-0-g30ef4aad21-amd64
lts-1.93-enterprise-search-contention

victoria-metrics-20240425-150852-tags-v1.101.0-enterprise-0-g718138c64
v1.101.0-enterprise

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 564463259a)
2024-04-30 10:29:56 +02:00
Zakhar Bessarab
88677f179c
dashboards/victoria-metrics-single: allow selecting multiple instance values (#5870)
Allowing to select multiple instance IPs makes it much easier to view
metrics for longer periods of time in dynamic environments such as
Kubernetes. In k8s update will also cause IP to change making it harder
to use dashboard to check the status.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5869

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6b493582da)
2024-04-30 10:29:54 +02:00
Github Actions
954abe78a3
Automatic update operator docs from VictoriaMetrics/operator@bbf847c (#6189)
(cherry picked from commit 3a669b4b87)
2024-04-30 10:29:29 +02:00
hagen1778
310d100ed7
docs/CHANGELOG.md: cut v1.101.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5334f0c2ce)
2024-04-25 16:25:38 +02:00
hagen1778
5b82f615ff
docs: add change for c0e4ccb7b5
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a62551f773)
2024-04-25 16:02:57 +02:00
Hui Wang
7fdea4b31c
app/vmselect: implement cmd-line flags -search.disableImplicitConversions and -search.logImplicitConversions (#6180)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338
support disable or log [implicit
conversions](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions)
for subquery with cmd-line flags `-search.disableImplicitConversion` and
`-search.logImplicitConversion`

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit dd0d2c77c8)
2024-04-25 13:08:05 +02:00
Yury Molodov
14c0c06526
vmui: improve error message for server response issues (#6177)
Updates error messages for better clarity and guidance on server
response issues.

(cherry picked from commit 57b7d16259)
2024-04-25 13:08:02 +02:00
Yury Molodov
669cbcb92e
vmui: trigger auto-suggestion at any cursor position (#6155)
- Implemented auto-suggestion triggers for mid-string cursor positions
in vmui.
- Improved the suggestion list positioning to appear directly beneath
the active text editing area.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5864
(cherry picked from commit 6193fa3dcf)
2024-04-25 13:08:00 +02:00
hagen1778
d4e901e212
Revert "docs: removed code-style highlighting for commanad-line flags of VM components (#6147)"
This reverts commit 9bedbcfa2f.

(cherry picked from commit 6aaf1768f4)
2024-04-25 13:07:58 +02:00
hagen1778
6f6e9e1e51
docs/changelog: mention downsampling fixes for ENT version of VM
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2d9bbe1934)
2024-04-24 17:08:28 +02:00
hagen1778
59b3f21708
Revert "app/vmbackup: introduce new flag type URL (#6152)"
This reverts commit 029060af60.

(cherry picked from commit 679844feaf)
2024-04-24 17:08:26 +02:00
Roman Khavronenko
ff73b66182
app/vmbackup: introduce new flag type URL (#6152)
The new flag type is supposed to be used for specifying URL values which
could contain sensitive information such as auth tokens in GET params or
HTTP basic authentication.

The URL flag also allows loading its value from files if `file://`
prefix is specified. As example, the new flag type was used in
app/vmbackup as it requires specifying `authKey` param for making the
snapshot.

See related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5973

Thanks to @wasim-nihal for initial implementation
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6060

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 029060af60)
2024-04-24 17:08:24 +02:00
Github Actions
ff0a1eb3e6
Automatic update operator docs from VictoriaMetrics/operator@6e1a876 (#6179)
(cherry picked from commit a14acd4fcc)
2024-04-24 17:08:21 +02:00
Artem Navoiev
4f07d32548
docs: vmalert remove new lines in configuration section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 592a9fe9f2)
2024-04-24 17:08:19 +02:00
Artem Navoiev
f22cfdc616
docs: vmalert remove new lines in configuration section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit aeba3f39db)
2024-04-24 17:08:18 +02:00
hagen1778
fcff6cc12a
docs/changelog: mention icnreased CPU usage in 1.100.1 release for ENT distributions
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 22cfab1ea2)
2024-04-24 17:08:13 +02:00
Github Actions
f2897fb859
Automatic update operator docs from VictoriaMetrics/operator@e343a8e (#6168)
(cherry picked from commit 301bd387d4)
2024-04-23 14:52:33 +02:00
Andrii Chubatiuk
29f9b8ef5d
deployment/docker: allow cross-platform building on arm64 platform (#6158)
Added x86_64 libraries to allow building cross-platform images on arm64

(cherry picked from commit 7958f38864)
2024-04-23 14:52:30 +02:00
Roman Khavronenko
2566e19306
app/vmalert: fix links with anchors in vmalert's UI (#6146)
Starting from v1.99.0 vmalert could ignore anchors pointing to specific
rule groups if `search` param was present in URL.
This change makes anchors compatible with `search` param in UI.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f487c7090)
2024-04-22 15:05:23 +02:00
Github Actions
d1ee860d46
Automatic update operator docs from VictoriaMetrics/operator@7e2ba6f (#6165)
(cherry picked from commit 6dbb4c1671)
2024-04-22 15:05:17 +02:00
Denys Holius
cad2cb2b56
docs: removed code-style highlighting for commanad-line flags of VM components (#6147)
Using `sh` or `console` formatting doesn't do word-breaking on render. This makes flags description
harder to read, as users need to scroll the web page horizontally.
Removing the formatting renders the description with normal word-breaking.

(cherry picked from commit 9bedbcfa2f)
2024-04-22 15:04:49 +02:00
hagen1778
342290275e
app/streamaggr: follow-up after c0e4ccb7b5
* rm vmagent mentions from vminsert flags
* improve documentation wording, add links to related sections
* mention `ignore_first_intervals` in the stream aggr options
* update flags description
* add basic test for config parsing validation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit bae3874e6a)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-22 14:39:23 +02:00
Andrii Chubatiuk
131367fb59
lib/streamaggr: add option to ignore first N aggregation intervals (#6137)
Stream aggregation may yield inaccurate results if it processes incomplete data.
This issue can arise when data is sourced from clients that maintain a queue of unsent data, such as Prometheus or vmagent.
 If the queue isn't fully cleared within the aggregation interval, only a portion of the time series may be included in that period, leading to distorted calculations.
To mitigate this we add an option to ignore first N aggregation intervals. It is expected, that client queues
will be cleared during the time while aggregation ignores first N intervals and all subsequent aggregations
will be correct.

(cherry picked from commit c0e4ccb7b5)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-22 14:34:36 +02:00
Aliaksandr Valialkin
ff122798b1
docs/MetricsQL.md: add links to raw samples chapter at https://docs.victoriametrics.com/keyconcepts/ 2024-04-21 00:22:03 +02:00
Aliaksandr Valialkin
946757fa5e
docs/CONTRIBUTING.md: small re-wording 2024-04-21 00:08:02 +02:00
Aliaksandr Valialkin
c254a889f7
docs: remove duplicate information regarding contribution guidelines to VictoriaMetrics
Substitute duplicate information with the link to https://docs.victoriametrics.com/contributing/ across all the docs.
2024-04-21 00:01:33 +02:00
Aliaksandr Valialkin
aa496d8827
docs/CONTRIBUTING.md: more formatting fixes and clarifications 2024-04-20 23:49:13 +02:00
Aliaksandr Valialkin
59648c880a
docs/CONTRIBUTING.md: small corrections after a402847eb6 2024-04-20 23:22:04 +02:00
Aliaksandr Valialkin
b16dbab137
Move CONTRIBUTING.md to docs/
It is better from visibility PoV if the CONTRIBUTING.md file is visible at https://docs.victoriametrics.com/contributing/ .
This page can be indexed by search engines and searched by our users later.
It is also easier to provide a link to this page now comparing to the old link, which is much harder to remember:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md

While at it, syncrhonize docs/CONTRIBUTING.md with .github/PULL_REQUEST_TEMPLATE/pull_request_template.md ,
so they do not contain duplicate information, which can be outdated over time. Now all the relevant information
is located at docs/CONTRIBUTING.md, while .github/PULL_REQUEST_TEMPLATE/pull_request_template.md refers this doc.

This is a follow-up for c006db1798

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6040
2024-04-20 23:12:53 +02:00
Bartosz Fenski
b36661539f
typo (#6159)
s/infromation/information/
2024-04-20 20:49:16 +02:00
Github Actions
ac7746d7fa
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@5b8a0ba (#6156) 2024-04-20 20:48:44 +02:00
Aliaksandr Valialkin
fba3c10ed1
app/vmselect/promql: add support for matching against multiple numeric constants via q == (c1,...,cN) and q != (c1,...,cN) syntax 2024-04-19 17:57:09 +02:00
Aliaksandr Valialkin
5504b6f2bd
docs/CHANGELOG.md: document v1.97.4 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.4
2024-04-19 13:10:37 +02:00
Aliaksandr Valialkin
a40a900b43
docs/CHANGELOG.md: document v1.93.14 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14
2024-04-19 12:52:38 +02:00
Aliaksandr Valialkin
a674029111
docs/LTS-releases.md: change the latest v1.93.x LTS release from v1.93.13 to v1.93.14
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14
2024-04-19 12:52:36 +02:00
Aliaksandr Valialkin
5b29be1f4d
app/vmagent/remotewrite: add support for replication additionally to sharding when both -remoteWrite.shardByURL and -remoteWrite.shardByURLReplicas=RF command-line flags are set
This allows setting up data replication among failure domains if the replication factor is smaller than the number of failure domains.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054

See https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages
2024-04-19 11:37:04 +02:00
Github Actions
0fde6f4b8f
Automatic update operator docs from VictoriaMetrics/operator@3d7bcba (#6149) 2024-04-19 11:36:38 +02:00
Github Actions
66f3f90f74
Automatic update operator docs from VictoriaMetrics/operator@781dbba (#6143) 2024-04-19 11:35:58 +02:00
Github Actions
a0b1c95787
Automatic update operator docs from VictoriaMetrics/operator@731de81 (#6139)
Co-authored-by: Alexander Marshalov <_@marshalov.org>
2024-04-19 11:35:21 +02:00
Github Actions
3cb3a9c950
Automatic update operator docs from VictoriaMetrics/operator@7f6c6d8 (#6138) 2024-04-19 11:34:50 +02:00
Hui Wang
e0d47ab6af
vmalert: avoid blocking APIs when alerting rule uses template functio… (#6129)
* vmalert: avoid blocking APIs when alerting rule uses template function `query`

* app/vmalert: small refactoring

* simplify labels and templates expanding
* simplify `newAlert` interface
* fix `TestGroupStart` which mistakenly skipped annotations
and response labels check

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* reduce alerts lock time when restore

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-19 11:30:40 +02:00
Aliaksandr Valialkin
f3967737b0
docs/Cluster-VictoriaMetrics.md: typo fix after dadeb6620a: folling -> following 2024-04-19 00:24:52 +02:00
Aliaksandr Valialkin
90794e84bc
app/vmselect/netstorage: add support for the ability to set cross-group replication factor at vmselect
The cross-group replication factor can be set via `-globalReplicationFactor` command-line flag at vmselect.
In this case vmselect continues returning full responses if up to globalReplicationFactor-1 groups are unavailable.

See https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect for details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054
2024-04-19 00:16:04 +02:00
Aliaksandr Valialkin
e2bc49534b
all: replace old https://docs.victoriametrics.com/PerTenantStatistic.html url with the new one - https://docs.victoriametrics.com/pertenantstatistic/ 2024-04-18 03:36:23 +02:00
Aliaksandr Valialkin
4f1eb03441
all: replace old https://docs.victoriametrics.com/Quick-Start.html url with the new one - https://docs.victoriametrics.com/quick-start/ 2024-04-18 03:33:01 +02:00
Aliaksandr Valialkin
b1477aa2be
all: replace old https://docs.victoriametrics.com/scrape_config_examples.html url with the new one - https://docs.victoriametrics.com/scrape_config_examples/ 2024-04-18 03:29:54 +02:00
Aliaksandr Valialkin
87641fa7e7
all: replace old https://docs.victoriametrics.com/Troubleshooting.html url with the new one - https://docs.victoriametrics.com/troubleshooting/ 2024-04-18 03:27:18 +02:00
Aliaksandr Valialkin
0c87cbe338
all: replace old https://docs.victoriametrics.com/relabeling.html url with the new one - https://docs.victoriametrics.com/relabeling/ 2024-04-18 03:23:15 +02:00
Aliaksandr Valialkin
10f3540bc6
all: replace old https://docs.victoriametrics.com/Articles.html url with the new one - https://docs.victoriametrics.com/articles/ 2024-04-18 03:20:13 +02:00
Aliaksandr Valialkin
987c4a2da5
all: replace old https://docs.victoriametrics.com/CaseStudies.html url with the new one - https://docs.victoriametrics.com/casestudies/ 2024-04-18 03:16:54 +02:00
Aliaksandr Valialkin
b933001d2d
all: replace old https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/single-server-victoriametrics/ 2024-04-18 03:11:49 +02:00
Aliaksandr Valialkin
b0129fab30
all: replace old https://docs.victoriametrics.com/url-examples.html url with the new one - https://docs.victoriametrics.com/url-examples/ 2024-04-18 03:02:11 +02:00
Aliaksandr Valialkin
a21d1fcf57
all: replace old https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/cluster-victoriametrics/ 2024-04-18 02:56:28 +02:00
Aliaksandr Valialkin
819569beda
all: replace old https://docs.victoriametrics.com/FAQ.html url with the new one - https://docs.victoriametrics.com/faq/ 2024-04-18 02:46:30 +02:00
Aliaksandr Valialkin
ed5744f072
all: replace old https://docs.victoriametrics.com/CHANGELOG_YYYY.html url with the new one - https://docs.victoriametrics.com/changelog_YYYY/ 2024-04-18 02:42:16 +02:00
Aliaksandr Valialkin
9fa1e5d843
all: replace old https://docs.victoriametrics.com/CHANGELOG.html url with the new one - https://docs.victoriametrics.com/changelog/ 2024-04-18 02:39:09 +02:00
Aliaksandr Valialkin
baf5c8d6d0
all: replace old https://docs.victoriametrics.com/keyConcepts.html url with the new one - https://docs.victoriametrics.com/keyconcepts/ 2024-04-18 02:34:09 +02:00
Aliaksandr Valialkin
513e69c55e
all: replace old https://docs.victoriametrics.com/sd_configs.html url with the new one - https://docs.victoriametrics.com/sd_configs/ 2024-04-18 02:28:26 +02:00
Aliaksandr Valialkin
728bcf0585
all: replace old https://docs.victoriametrics.com/stream-aggregation.html url with the new one - https://docs.victoriametrics.com/stream-aggregation/ 2024-04-18 02:20:00 +02:00
Aliaksandr Valialkin
64938732e3
all: replace old https://docs.victoriametrics.com/MetricsQL.html url with the new one - https://docs.victoriametrics.com/metricsql/ 2024-04-18 02:15:33 +02:00
Aliaksandr Valialkin
2ec18a6634
all: replace old https://docs.victoriametrics.com/vmgateway.html url with the new one - https://docs.victoriametrics.com/vmgateway/ 2024-04-18 02:08:53 +02:00
Aliaksandr Valialkin
a501e237d2
all: replace old https://docs.victoriametrics.com/vmbackupmanager.html url with the new one - https://docs.victoriametrics.com/vmbackupmanager/ 2024-04-18 02:04:39 +02:00
Aliaksandr Valialkin
a85602bb2d
all: replace old https://docs.victoriametrics.com/vmrestore.html url with the new one - https://docs.victoriametrics.com/vmrestore/ 2024-04-18 02:01:18 +02:00
Aliaksandr Valialkin
6fe21520ce
all: replace old https://docs.victoriametrics.com/vmbackup.html url with the new one - https://docs.victoriametrics.com/vmbackup/ 2024-04-18 01:58:00 +02:00
Aliaksandr Valialkin
c7866f3e19
all: replace old https://docs.victoriametrics.com/vmctl.html url with the new one - https://docs.victoriametrics.com/vmctl/ 2024-04-18 01:54:16 +02:00
Aliaksandr Valialkin
728aceebfe
all: replace old https://docs.victoriametrics.com/vmauth.html url with the new one - https://docs.victoriametrics.com/vmauth/ 2024-04-18 01:50:16 +02:00
Aliaksandr Valialkin
a99005eff6
all: replace old https://docs.victoriametrics.com/vmalert.html url with the new one - https://docs.victoriametrics.com/vmalert/ 2024-04-18 01:44:54 +02:00
Aliaksandr Valialkin
0211a04a52
all: replace the outdated url https://docs.victoriametrics.com/vmagent.html with the new one - https://docs.victoriametrics.com/vmagent/ 2024-04-18 01:32:57 +02:00
Aliaksandr Valialkin
e615bcced7
lib/storage: improve performance for /api/v1/label/labelName/values when match[] contains only a single filter on labelName
This speeds up auto-suggestion for metric names in VMUI and Grafana, which use the following query in this case:

  /api/v1/label/__name__/values?match[]={__name__=~"*.some_value.*"}

When the user types `some_value` in the query input field.
2024-04-18 01:18:03 +02:00
Aliaksandr Valialkin
164032cd9b
lib/httpserver: add support for automatic issuing of TLS certificates via Lets Encrypt service
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949
2024-04-17 23:53:51 +02:00
Aliaksandr Valialkin
30c96ba8d7
app/{vminsert,vmselect}: support for srv+addr scheme for specifying DNS SRV addresses at -storageNode flag
The new scheme is consistent with SRV urls introduced at b426d10847 and dc326f70b4

Deprecte the old scheme: `dns+srv:addr` by removing it from the docs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053
2024-04-17 23:15:05 +02:00
Aliaksandr Valialkin
9f33b3760d
docs/vmagent.md: typo fixes after dc326f70b4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053
2024-04-17 21:13:01 +02:00
Aliaksandr Valialkin
284d99e269
app/vmagent: support for DNS SRV urls at -remoteWrite.url, scrape target urls and service discovery urls
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053
2024-04-17 20:56:23 +02:00
Aliaksandr Valialkin
e627810146
app/vmauth: add support for configuring backends via DNS SRV urls 2024-04-17 20:56:21 +02:00
Aliaksandr Valialkin
6cc80ab621
app/vmauth: add support for client TLS sertificates for backend requests over https
While at it, also add support for TLS ServerName for backend requests over https
2024-04-17 17:13:12 +02:00
Aliaksandr Valialkin
eb870d9de3
docs/vmauth.md: add Authorization and Routing chapters 2024-04-17 16:17:21 +02:00
Aliaksandr Valialkin
6cdf01297e
app/vmauth: follow-up for b155b20de4
- Use exact matching by default for the query arg value provided via arg=value syntax at src_query_args.
  Regex matching can be enabled by using =~ instead of = . For example, arg=~regex.
  This ensures that the exact matching works as expected without the need to escape special regex chars.

- Add helper functions for creating QueryArg, Header and Regex structs in tests.
  This improves maintainability of the tests.

- Remove url.QueryUnescape() call on the url in TestCreateTargetURLSuccess(), since this is bogus approach.
  The url.QueryUnescape() must be applied to individual query args, and it mustn't be applied to the whole url,
  since in this case it may perform invalid unescaping in the context of the url, or make the resulting url invalid.

While at it, properly marshal all the fields inside UserInfo config to yaml in tests.
Previously Header and QueryArg structs were improperly marshaled because the custom MarshalYAML
is called only on pointers to Header and QueryArg structs. This improves test coverage.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6115
2024-04-17 14:53:20 +02:00
Aliaksandr Valialkin
54c9f9c5ea
docs/Cluster-VictoriaMetrics.md: clarify per-node workload increase when one of vmstorage node is unavailable in the cluster
The docs update is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6099#issuecomment-2060856417
2024-04-17 14:52:49 +02:00
Yury Molodov
e4d6ded0f5
vmui: preserve Select value if it matches list on blur (#6101) 2024-04-17 14:52:30 +02:00
Roman Khavronenko
256c561005
app/vmauth: support regex matching in src_query_args (#6115)
Support regex matching when routing incoming requests based on HTTP query args
via `src_query_args` option at `url_map`.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-17 14:52:02 +02:00
Aliaksandr Valialkin
c6f0a37125
docs/Cluster-VictoriaMetrics.md: remove incorrect information about the increase of workload on the remaining vmstorage nodes when one vmstorage node is unavailable
If one out of 5 vmstorage nodes is unavailable, then the remaining 4 vmstorage nodes will recieve 1/5=20% increase of workload, not 5%.
If one out of 10 vmstorage nodes is unavailable, then the remaining 9 vmstorage nodes will receive 1/10=10% increase of workload, not 1%.

This is a follow-up for 458338afa5

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6099
2024-04-17 14:49:59 +02:00
Zakhar Bessarab
3f618c0485
app/vmauth: do not increment backend_errors when hitting concurrency limit (#6078)
* app/vmauth: do not increment backend_errors when hitting concurrency limit

Previously, both "vmauth_concurrent_requests_limit_reached_total" and "vmauth_user_request_backend_errors_total" were incremented.
This was based on the assumption that if concurrency limit is hit the backend must be failing to handle the request thus meaning an error.

This assumption does not work in case the endpoint can be overloaded by the misbehaving client sending too many requests within the timeframe.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-17 12:34:33 +02:00
Github Actions
60a447bb76
Automatic update operator docs from VictoriaMetrics/operator@99fbc98 (#6122) 2024-04-17 12:15:39 +02:00
Aliaksandr Valialkin
2347699126
docs/CHANGELOG.md: move the description for the bugfix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110 from v1.100.1 to the tip section
The bugfix isn't included in v1.100.1 release.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6111
This is a follow-up for 7308bad777
2024-04-17 12:13:01 +02:00
Zakhar Bessarab
691181867e
docs/cluster: update cluster resizing info (#6099)
* docs/cluster: update cluster resizing info

- add example of resources distribution
- add info about how to handle uneven disk usage after adding a new storage node

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 458338afa5)
2024-04-17 09:40:36 +02:00
Github Actions
a8444cb54b
Automatic update operator docs from VictoriaMetrics/operator@13f6dac (#6119)
(cherry picked from commit aaa18e565d)
2024-04-17 09:40:34 +02:00
hagen1778
3bddb10af8
docs: mention HTTP sink configuration example for Vector
Follow-up 16eeb4e

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4f55aa29db)
2024-04-17 09:40:32 +02:00
Devin Buhl
a09da7a42c
victorialogs: mention vector supports http/json stream (#6114)
https://github.com/vectordotdev/vector/issues/18883#issuecomment-1771424716
(cherry picked from commit 16eeb4eb33)
2024-04-17 09:40:27 +02:00
yudrywet
d89229c838
chore: fix some typos in comments (#6103)
Signed-off-by: yudrywet <yudeyao@yeah.net>

(cherry picked from commit 43835704b7)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-16 09:58:30 +02:00
Alexander Marshalov
c61eb6d620
vmalert: support any status code from the range 200-299 from alertmanager as successful (#6111)
* any status code from the range 200-299 from alertmanager to vmalert is not considered an error from now on (#6110)

* add changelog

(cherry picked from commit 7308bad777)
2024-04-16 09:58:09 +02:00
Github Actions
20f16aed1a
Automatic update operator docs from VictoriaMetrics/operator@cf48a99 (#6113)
(cherry picked from commit 7db8ba41e7)
2024-04-16 09:58:07 +02:00
Dmytro Kozlov
097cc8a2a8
docs: fix typo in the curl command (#6109)
(cherry picked from commit 7b20de4674)
2024-04-16 09:58:05 +02:00
Github Actions
4859f5eeed
Automatic update operator docs from VictoriaMetrics/operator@73a1996 (#6100)
(cherry picked from commit cba2f6dce1)
2024-04-16 09:57:44 +02:00
hagen1778
b81a66ceea
docs: add 1.100.1 release date
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit e39a1a98f5)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-16 09:57:41 +02:00
Hui Wang
9c0da1ef3d
deployment: bump victoriametrics components to v1.100.1 (#6096) 2024-04-12 12:19:21 +02:00
Aliaksandr Valialkin
4179ebdce7
deployment: upgrade VictoriaLogs from v0.5.1-victorialogs to v0.5.2-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.2-victorialogs
2024-04-11 09:53:36 +02:00
Aliaksandr Valialkin
14448966d9
docs/VictoriaLogs/CHANGELOG.md: cut v0.5.2-victorialogs release 2024-04-11 09:42:22 +02:00
Aliaksandr Valialkin
67af029b7f
docs/VictoriaLogs/CHANGELOG.md: document the bugfix at 2205de2391
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6042
2024-04-11 09:42:20 +02:00
Aliaksandr Valialkin
8cb768a7a5
docs/CHANGELOG.md: cut v1.100.1 release 2024-04-11 09:35:27 +02:00
Aliaksandr Valialkin
14a6dee206
docs/CHANGELOG.md: mention that the bug with incorrect registration of new entries in the IndexDB has been introduced in v1.99.0 2024-04-11 09:35:26 +02:00
Zakhar Bessarab
da8bb106aa
lib/mergeset: fix flushing incorrect set of inmemoryBlocks (#6089)
Follow-up for bace9a2501

Related:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6069
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-04-11 09:26:48 +02:00
Github Actions
26ed8d0571
Automatic update operator docs from VictoriaMetrics/operator@17082f0 (#6090) 2024-04-11 09:15:21 +02:00
Zakhar Bessarab
956ff263fc
docs/changelog: add entry for bd398974334678a29972475b0566e0bd2434d197
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-04-11 09:12:12 +02:00
7c4c6f7ee6
docs: fix a typo in docs/operator/resources/vmalert.md (#6088) 2024-04-11 09:09:26 +02:00
Aliaksandr Valialkin
e46150c3bf
docs/CHANGELOG.md: remove description of the reverted change at c79bf3925c
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985
2024-04-04 16:56:04 +03:00
Aliaksandr Valialkin
b8d5fe0584
deployment/docker: update VictoriaLogs tag from v0.5.0-victorialogs to v0.5.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.1-victorialogs
2024-04-04 16:52:06 +03:00
Aliaksandr Valialkin
a549fc2941
docs/VictoriaLogs/CHANGELOG.md: cut v0.5.1-victorialogs release 2024-04-04 16:41:38 +03:00
Aliaksandr Valialkin
c55afb9a18
docs/VictoriaLogs/CHANGELOG.md: document the fix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5927
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920

This is a follow-up for 46fd0ed693
2024-04-04 16:39:39 +03:00
Aliaksandr Valialkin
fe28388c5d
deployment: update VictoriaMetrics docer image from v1.99.0 to v1.100.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.100.0
2024-04-04 15:29:12 +03:00
Aliaksandr Valialkin
18e521dbff
docs: change old url from https://docs.victoriametrics.com/enterprise.html to new url https://docs.victoriametrics.com/enterprise/ 2024-04-04 15:22:38 +03:00
Aliaksandr Valialkin
68a9bcaa1e
docs/Single-server-VictoriaMetrics.md: remove misleading filters word from the text, which recommends evaluating downsampling in VictoriaMetrics enterprise 2024-04-04 14:58:45 +03:00
Aliaksandr Valialkin
6cf8d2144e
docs: clarify that downsampling drops all the samples except the last one on every downsampling interval 2024-04-04 14:50:24 +03:00
Aliaksandr Valialkin
34687cfec4
docs/CHANGELOG.md: typo fix: remove duplicate samples word 2024-04-04 14:46:21 +03:00
Aliaksandr Valialkin
6332a1c447
docs/CHANGELOG.md: advise upgrading from v1.99.0 to v1.100.0 because of the issue at v1.99.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959
2024-04-04 14:41:31 +03:00
Aliaksandr Valialkin
283f4b96f6
docs/CHANGELOG.md: move custom backup interval feature to tip, since it wasnt included in v1.100.0 release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966
This is a follow-up for baa869742208f311d84e800ea527a7f07eb1ca18
2024-04-04 12:33:49 +03:00
Zakhar Bessarab
27f5eb38bb
Vmbackupmanager: add support of custom backup interval (#742)
* app/vmbackupmanager: add support of defining custom backup interval

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: update vmbackupmanager docs

Update docs after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-04-04 12:33:48 +03:00
Aliaksandr Valialkin
530d7b40bd
docs/CHANGELOG.md: add release date for v1.100.0 2024-04-04 12:28:31 +03:00
Aliaksandr Valialkin
ddb1b5b70f
docs/CHANGELOG.md: cut v1.100.0 2024-04-04 03:47:32 +03:00
Aliaksandr Valialkin
d61f6c89a1
app/{vmagent,vminsert}: accept Prometheus remote write protocol requests at /prometheus/api/v1/push additionally to /api/v1/push
This is a follow-up for 7ccdb57ea4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5990
2024-04-04 02:17:10 +03:00
nemobis
fdfdc9f8f1
Add note about final deduplication space needs (#5996)
Addresses #5975

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-04-04 02:09:12 +03:00
Yury Molodov
2d8777ac1e
vmui: fix step update on input blur in Firefox/Safari (#6034)
* vmui: fix step value application on input blur

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-04-04 01:53:15 +03:00
Yury Molodov
da642b56a1
vmui: fix trigger auto-suggestion (#6033)
* vmui: fix ui freeze on query paste #5923

* vmui: fix auto-suggestion trigger issue after whitespace char #5866

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-04-04 01:49:17 +03:00
Aliaksandr Valialkin
ecf3dfde9a
deployment: update Go builder from 1.22.1 to 1.22.2
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved
2024-04-04 01:44:58 +03:00
Aliaksandr Valialkin
ecd782c75e
app/vmagent: follow-up for b3b29ba6ac
- Automatically reload changed TLS root CA pointed by -remoteWrite.tlsCAFile command-line flag
- Automatically reload changed TLS root CA configured via oauth2.tsl_config.ca_file option at -promscrape.config
- Document the change as a feature instead of a bug at docs/CHANGELOG.md
- Simplify the code at lib/promauth, which is responsible for reloading changed TLS root CA files.
- Simplify the usage of lib/promauth.Config.NewRoundTripper() - now it accepts the base http.Transport
  instead of a callback, which can change the internal http.Transport.
- Reuse the default tls config if lib/promauth.Config doesn't contain tls-specific configs.
  This should reduce memory usage a bit when tls isn't used for scraping big number of targets.
- Do not re-read TLS root CA files on every processed request. Re-read them once per second.
  This should reduce CPU usage when scraping big number of targets over https.
- Do not store cert.pem and key.pem files in TestTLSConfigWithCertificatesFilesUpdate, since they can be loaded
  from byte slices via crypto/tls.X509KeyPair().
- Remove obsolete comparisons of string representations for authConfig and proxyAuthConfig at areEqualScrapeConfigs().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5725
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171
2024-04-04 01:26:38 +03:00
Aliaksandr Valialkin
1c6dbe5714
docs/CHANGELOG.md: add - in front of -logInvalidAuthTokens command-line flag in order to be consistent with command-line flag naming
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029
2024-04-04 00:42:37 +03:00
Zakhar Bessarab
bb2d029526
vmgateway: add an ability to log invalid auth tokens (#743)
* app/vmgateway: add an ability to log invalid auth tokens

This is useful for debugging to make it easier for user to find issues in token contents.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add info about new vmgateway flag

- add changelog entry
- add info about logInvalidAuthTokens flag

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmgateway/filters/auth: improve reject reason visibility

Explicitly return a rejection reason for request when "logInvalidAuthTokens" is enabled.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-04-04 00:42:09 +03:00
Aliaksandr Valialkin
2be627f1fe
app/vmalert: document that -rule.stripFilePath command-line flag is available only in enterprise version of vmalert 2024-04-04 00:41:35 +03:00
Thomas
7526f1d3a5
chore(docs): fix vmalertmanager typo (#6056)
Fixes: #6055

Signed-off-by: Thomas Way <thomas@6f.io>
Co-authored-by: Alexander Marshalov <_@marshalov.org>
2024-04-04 00:39:53 +03:00
Github Actions
ecee306e7a
Automatic update operator docs from VictoriaMetrics/operator@92cdca3 (#6052) 2024-04-04 00:39:19 +03:00
Zakhar Bessarab
da4352fe7b
lib/{promauth,promscrape}: automatically refresh root CA certificates after changes on disk (#5725)
* lib/{promauth,promscrape}: automatically refresh root CA certificates after changes on disk

Added a custom `http.RoundTripper` implementation which checks for root CA content changes and updates `tls.Config` used by `http.RoundTripper` after detecting CA change.

Client certificate changes are not tracked by this implementation since `tls.Config` already supports passing certificate dynamically by overriding `tls.Config.GetClientCertificate`.

This change implements dynamic reload of root CA only for streaming client used for scraping. Blocking client (`fasthttp.HostClient`) does not support using custom transport so can't use this implementation.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promauth/config: update NewRoundTripper API

Update API to allow user to update only parameters required for transport.

Add warning log when reloading Root CA failed.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promauth/config: fix mutex acquire logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promauth/config: replace RWMutex with regular mutex to simplify the code

- remove additional mutex used for getRootCABytes - require callee to use mutex
- replace RWMutex with regular mutex

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promauth/config: refactor

- hold the mutex lock to avoid round tripper being re-created twice
- move recreation logic into separate func to simplify the code

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-04-04 00:34:43 +03:00
Aliaksandr Valialkin
2e1c8bb436
docs/CHANGELOG.md: typo fix: resonses -> responses 2024-04-03 03:20:09 +03:00
Aliaksandr Valialkin
62c1545f97
docs/CHANGELOG.md: remove duplicate line left after 1e0a079143 2024-04-03 03:19:48 +03:00
hagen1778
6a3bf16475
docs: update vmalert troubleshooting docs
* rm recommendation to keep look-behind window empty, as it is not correct
* mention the change of default value for `-search.latencyOffset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-03 03:15:11 +03:00
Aliaksandr Valialkin
bb9bb600b3
lib/protoparser/opentelemetry: follow-up after 47892b4a4c
- Rename -opentelemetry.sanitizeMetrics command-line flag to more clear -opentelemetry.usePrometheusNaming
- Clarify the description of the change at docs/CHANGELOG.md
- Rename promrelabel.SanitizeLabelNameParts to more clear promrelabel.SplitMetricNameToTokens
- Properly split metric names at '_' char in promerlabel.SplitMetricNameToTokens.
- Add tests for various edge cases for Prometheus metric names' normalization
  according to the code at b865505850/pkg/translator/prometheus/normalize_name.go
- Extract the code responsible for Prometheus metric names' normalization into a separate file (santize.go)

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6035
2024-04-03 03:09:52 +03:00
Aliaksandr Valialkin
a5f65756f8
docs: follow-up for ac9c2a796f
Remove description for -search.maxExportDuration and -search.maxStatusRequestDuration command-line flags
from the 'Resource usage limits' chapter, since these flags are rarely used for limiting resource usage
and they are already documented in the 'List of command-line flags' chapter.
2024-04-03 03:08:27 +03:00
Aliaksandr Valialkin
5874189797
app/vmselect/graphite: follow-up for 23ab865035
- Fix docs for new functions at app/vmselect/graphite/functions.json
- Properly drain series lists on errors in aggregateSeriesListsGeneric() and aggregateSeriesList()
- Add links to docs for the added functions at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5809
2024-04-03 03:07:15 +03:00
Aliaksandr Valialkin
f1b5784432
app/vmauth: add ability to authorize via any opaque HTTP request header value
This can be done via `auth_token` option at -auth.config - see https://docs.victoriametrics.com/vmauth/#auth-config
2024-04-03 02:52:39 +03:00
Aliaksandr Valialkin
2e9ee89cf1
app/vmauth: add ability to read auth tokens from multiple http request headers
This is needed for VictoriaMetrics Cloud, where the same token could be passed either
via Authorization or via X-Amz-Firehose-Access-Key header - see 4487dac30b (r140500722)

This is a follow-up for 4487dac30b

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-03 02:52:20 +03:00
Aliaksandr Valialkin
456180b9d9
app/vmauth: follow-up for bc90f4aae6
- Allow specifying only a single HTTP header for reading auth tokens via -httpAuthHeader command-line flag.
  This is better from security PoV, since this prevents from accidental reading of auth token from undesired
  HTTP header. By default the -httpAuthHeader equals to Authorization. When it is overridden, then
  auth token isn't read from Authorization header - it is read only from the specified header.

- Document the -httpAuthHeader command-line flag at https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-03 02:51:28 +03:00
Aliaksandr Valialkin
faa2ba828a
app/vmagent: simplify code after 509df44d03
- Simplify the code in order to improve its maintenance
- Properly pass tenant ID when processing multi-tenant opentelemetry request at vmagent

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6016
2024-04-03 02:50:46 +03:00
Fred Navruzov
28f7b2218a
docs/vmanomaly: typos fix (#6047) 2024-04-03 02:45:35 +03:00
Fred Navruzov
d83e05a3e9
docs/vmanomaly: v1.12 updates & fixes (#6046)
* docs/vmanomaly: v1.12.0 & link updates

* add autotuned description to model section

* - update refs of vmanomaly on enterprise and vmalert pages
- add diagrams for model types
- update self-monitoring section

* - fix typos
- remove .index.html from links
2024-04-03 02:44:58 +03:00
Aliaksandr Valialkin
e51190a34c
Revert "app/vmselect: make vmselect resilient to absence of cache folder (#5987)"
This reverts commit cb23685681.

Reason for revert: the "fix" may hide programming bugs related to incorrect creation of folders
before their use. This may complicate detecting and fixing such bugs in the future.

There are the following fixes for the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985 :
- To configure the OS to do not drop data from the system-wide temporary directory (aka /tmp).
- To run VictoriaMetrics with -cacheDataPath command-line flag, which points to the directory,
  which cannot be removed automatically by the OS.

The case when the user accidentally deletes the directory with some files created by VictoriaMetrics
shouldn't be considered as expected, so VictoriaMetrics shouldn't try resolving this case automatically.
It is much better from operation and debuggability PoV is to crash with the clear `directory doesn't exist` error
in this case.
2024-04-03 02:44:00 +03:00
Aliaksandr Valialkin
db6d5d6349
docs/CHANGELOG.md: mention that the bug with improper use of -search.maxExportDuration instead of -search.maxLabelsAPIDuration has been introduced in v1.99.0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5992
This is a follow-up for bc79f7196d
2024-04-03 02:43:40 +03:00
Aliaksandr Valialkin
83652eea1d
docs/VictoriaLogs/CHANGELOG.md: improve the description of the bugfix from 43b5d8bc7a, so it can be googled by users 2024-04-03 02:43:21 +03:00
Aliaksandr Valialkin
7edb5f77f1
app/vmagent: properly shutdown when -maxIngestionRate limit is reached
The remotewrite.Stop() expects that there are no pending calls to TryPush().
This means that the ingestionRateLimiter.Register() must be unblocked inside TryPush() when calling remotewrite.Stop().
Provide remotewrite.StopIngestionRateLimiter() function for unblocking the rate limiter before calling the remotewrite.Stop().

While at it, move the rate limiter into lib/ratelimiter package, since it has two users.
Also move the description of the feature to the correct place at docs/CHANGELOG.md.
Also cross-reference -remoteWrite.rateLimit and -maxIngestionRate command-line flags.

This is a follow-up for 02bccd1eb9
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900
2024-04-03 02:41:11 +03:00
Aliaksandr Valialkin
6719415750
docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md: user proper image paths according to docs/assets/README.md 2024-04-03 02:40:51 +03:00
Aliaksandr Valialkin
7d98632c9d
docs/managed-victoriametrics: use proper names for the linked images according to docs/assets/README.md
This is a follow-up for db3709c87d

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5989
2024-04-03 02:40:31 +03:00
Aliaksandr Valialkin
eff9676508
docs/CHANGELOG.md: remove the update notes regarding converting custom HTTP header keys to canonical form
Custom HTTP headers are set via net/http.Header.Set or net/http.Header.Add functions.
These functions always convert header keys to canonical form. So the change at b577413d3b
isn't visible to users of VictoriaMetrics components.

There is no need in documenting this change at docs/CHANGELOG.md, since it doesn't give any useful information to users.

This is a follow-up for e6dd52b04c
2024-04-03 02:39:08 +03:00
Zakhar Bessarab
3d15a31c6d
lib/storage: add ability to use downsampling for the given series filter (#733)
* lib/storage: add ability to use downsampling for the given series filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add information about downsampling filters

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: fix MetricsQL filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: treat missing downsampling filter as a bug

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/part_header: verify correctness of downsampling filters when opening partition

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: save only appliable rules in part metadata

Filter and save only rules which are appliable to partition based on MinTimestamp of stored data.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: update log messages for final dedup

Properly specify a reason of re-running deduplication for partition.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage: consistently use MaxTimestamp to determine deduplication/downsampling rules

Using MinTimestamp leads to applying downsampling to parts which are only partially covered by downsampling rule.
For example, partition covers range [1000-2000]. At t=2100 and rule offset 500 data with t=2100-500 => 1600 must be downsampled. The range check against MinTimestamp evaluates to true even though partition contains range which must not be downsampled - [1600:2000].

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Follow-up

- Apply the first matching downsampling period if multiple filters match the given time series.
  This allows fine-tuning the downsampling config for the specific needs.
- Take into account downsampling filters during search queries.
- Reduce the difference between community and enterprise branches. This should simplify further maintenance of these branches.
- Properly parse series filters with colons inside them.
- Document the feature at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-04-03 02:38:37 +03:00
Andrii Chubatiuk
0799834aaa
opentelemetry: added cmd flag to sanitize metric names (#6035) 2024-04-03 02:31:39 +03:00
Aliaksandr Valialkin
3f79e54a51
app/vmagent/remotewrite: follow-up for 166b97b8d0 and b6bd9a97a3
- Make the configuration more clear by accepting the list of ignored labels during sharding
  via a dedicated command-line flag - -remoteWrite.shardByURL.ignoreLabels.
  This prevents from overloading the meaning of -remoteWrite.shardByURL.labels command-line flag.

- Removed superfluous memory allocation per each processed sample if sharding by remote storage is enabled.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938
2024-04-03 00:50:48 +03:00
Dmytro Kozlov
111a8e5fd0
docs: describe timeout query argument (#6020)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-03 00:02:00 +03:00
Hui Wang
f410520cfa
docs: fix golangci-lint check (#6036)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-02 23:52:50 +03:00
Hui Wang
fdb6eb1071
vmalert: fix sending alert messages (#6028)
* vmalert: fix sending alert messages
1. fix `endsAt` field in messages that send to alertmanager, previously rule with small interval could never be triggered;
2. fix behavior of `-rule.resendDelay`, before it could prevent sending firing message when rule state is volatile.

* docs: update changelog notes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-02 23:51:06 +03:00
Aliaksandr Valialkin
93b389eb70
docs/MetricsQL.md: typo fix: outlier_iqr_over_time(memory_usage_bytes[1h]) triggers when memory_usage_bytes goes outside the usual value range for the last hour, not the last 24 hours
This is a follow-up for ea81f6fc36
2024-04-02 23:48:58 +03:00
hagen1778
467c1ba617
docs: mention new guide How to use OpenTelemetry metrics with VictoriaMetrics in docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-02 23:47:44 +03:00
Nikolay
2ee8e462f8
docs/opentelemetry: adds opentemetry get started guide (#5861)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
2024-04-02 23:47:06 +03:00
Github Actions
5d25fe206d
Automatic update operator docs from VictoriaMetrics/operator@c336013 (#6023) 2024-04-02 23:46:00 +03:00
Github Actions
ee95e9d14d
Automatic update operator docs from VictoriaMetrics/operator@ac29c88 (#6022) 2024-04-02 23:45:41 +03:00
Zakhar Bessarab
7c1ee69205
lib/storage/table: wait for merges to be completed when closing a table (#5965)
* lib/storage/table: properly wait for force merges to be completed during shutdown

Properly keep track of running background merges and wait for merges completion when closing the table.
Previously, force merge was not in sync with overall storage shutdown which could lead to holding ptw ref.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add changelog entry

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-04-02 21:25:30 +03:00
Andrii Chubatiuk
9c92cc2759
vmauth: support other auth header names besides Authorization (#6009) 2024-04-02 18:41:00 +03:00
Andrii Chubatiuk
914b23f1e8
app/{vmagent,vminsert}: fixed firehose response (#6016) 2024-04-02 18:03:12 +03:00
hagen1778
1c597e9dba
docs: follow-up after 623d257faf
623d257faf
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2e843a8ed9)
2024-03-29 14:30:25 +01:00
Jiekun
1e0a079143
app/vmalert: respect batch size limit for remote write on shutdown (#6039)
During shutdown period of vmalert, remotewrite client retrieve all pending time series from buffer queue, compose them into 1 batch and execute remote write.

This final batch may exceed the limit of -remoteWrite.maxBatchSize, and be rejected by the receiver (gateway, vmcluster or others).

This changes ensures that even during shutdown vmalert won't exceed the max batch size limit for remote write
destination.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025

(cherry picked from commit 623d257faf)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-29 14:30:24 +01:00
hagen1778
36c1ca9949
app/vmagent: follow-up 166b97b8d0
* add tests for sharding function
* update flags description
* add changelog note

166b97b8d0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b6bd9a97a3)
2024-03-29 14:10:17 +01:00
Roman Khavronenko
548bf31dd2
app/vmselect: make vmselect resilient to absence of cache folder (#5987)
vmselect uses a cache folder in file system for two purposes:
1. Storing rollup cache results on shutdown;
2. Storing temporary search results from vmstorage during query executions.

It could happen that cache folder is deleted accidentally by user, or by OS
during cleanup routines. This would cause vmselect to:
1. panic on /metrics call, because `MustGetFreeSpace` will fail;
2. return query error user, as it won't be able to store temporary search results.

The changes in this commit are the following:
1. Make `MustGetFreeSpace` to try re-creating the cache folder if it is missing;
2. Make vmselect to try re-creating the cache folder if it can't persist tmp search
results.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
(cherry picked from commit cb23685681)
2024-03-26 15:27:32 +01:00
Artem Navoiev
bf226879ec
add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit b08cbd0400)
2024-03-25 15:42:54 +01:00
Artem Navoiev
19a11c12d1
fix typo in kyiv city name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit b569fa0b2c)
2024-03-25 15:42:53 +01:00
Nikolay
60d64edd6e
app/vlselect: follow-up for 0514091948 (#6004)
removes println lines

(cherry picked from commit 43b5d8bc7a)
2024-03-25 15:42:53 +01:00
Alexander Marshalov
0b3744effa
[vmagent] added ingestion rate limiting with new flag -maxIngestionRate (#5900)
* [vmagent] added ingestion rate limiting with new flag `-maxIngestionRate`. This flag can be used to limit the number of samples ingested by vmagent per second. If the limit is exceeded, the ingestion rate will be throttled.

* fix changelog

* fix review comment

(cherry picked from commit 02bccd1eb9)
2024-03-25 15:42:52 +01:00
Nikolay
3ea0b87399
docs/managed: adds alertmanager configuration examples (#5989)
* docs/managed: adds alertmanager configuration examples

* apply review suggestions

(cherry picked from commit db3709c87d)
2024-03-25 15:42:52 +01:00
hagen1778
a1b666c35e
docs: add missing API version to VMSingle example
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 93a29fce4e)
2024-03-25 15:42:51 +01:00
hagen1778
2b05559293
docs: mention Query Analyzer in docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 21d9393c9e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-25 15:42:51 +01:00
Yury Molodov
30778e0d0f
vmui: fix the _time filter insertion for all queries in VictoriaLogs UI #5920 (#5927)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920
(cherry picked from commit 46fd0ed693)
2024-03-25 15:42:33 +01:00
Dmytro Kozlov
9817ecdcc9
app/vmctl: break explore phase in vm-native mode by time intervals
When `--vm-native-step-interval` is specified, explore phase will be executed
within specified intervals. Discovered metric names will be associated with
time intervals at which they were discovered. This suppose to reduce number
of requests vmctl makes per metric name since it will skip time intervals
when metric name didn't exist.

This should also reduce probability of exceeding complexity limits
for number of selected series in one request during explore phase.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5369
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f8b91186a)
2024-03-25 15:42:24 +01:00
hagen1778
9fcd34287b
docs: follow-up for 70eaa06f08
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit bc79f7196d)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-25 15:33:11 +01:00
Aliaksandr Valialkin
cd222d6502
lib/streamaggr: ignore out of order samples for last output
This is a follow-up for 6a465f6e29

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931
2024-03-18 01:03:58 +02:00
Aliaksandr Valialkin
eecc5e8463
lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search
The metricID->metricName entry can remain invisible for search for some time after registering new metricName.
This is expected condition. So wait for up to 60 seconds in the hope that the metricID->metricName
entry will become visible before deleting all the entries from indexdb, which are associated with the given metricID.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948

See also 20812008a7
2024-03-18 00:37:11 +02:00
Aliaksandr Valialkin
111d0aa2bf
app/{vmagent,vminsert}: add an ability to ignore input samples outside the current aggregation interval for stream aggregation
See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples
2024-03-17 23:30:46 +02:00
Aliaksandr Valialkin
e70b644f1f
lib/streamaggr: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs
Out of order samples may result in unexpected spikes for these outputs.
So it is better to ignore such samples.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931
2024-03-17 23:24:14 +02:00
Aliaksandr Valialkin
1f753a049a
lib/streamaggr: follow-up for 15e33d56f1
- Properly set pushSample.timestamp when flushing de-duplicated samples to stream aggregation
  This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931

- Re-classify this change as feature instead of bugfix at docs/CHANGELOG.md

- Verify de-duplication logic for samples with different timestamps

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5939
2024-03-17 23:23:57 +02:00
Aliaksandr Valialkin
c3dbcc7c0e
docs/CHANGELOG.md: clarify that -datasource.lookback commnad-line flag is no-op in the upcoming release
Document the solution - to switch to eval_delay option at group config.

This is a follow-up for e80b44f19d
2024-03-17 23:23:31 +02:00
Aliaksandr Valialkin
b481448c53
docs/CHANGELOG.md: document the bugfix from cb259116b4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802
2024-03-17 23:23:11 +02:00
Aliaksandr Valialkin
7c4d7dc6dd
lib/promauth: properly set Host header in requests to scrape targets.
The `Host` header must be set via net/http.Request.Host field, since net/http.Client
ignores this header if it is set via Request.Header.Set().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970
2024-03-17 23:22:54 +02:00
hagen1778
efddda945e
app/vmctl: fix arguments order in httputils.TLSConfig
follow-up after 9d5bf5ba5d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-17 23:22:08 +02:00
Github Actions
eb82f329cb
Automatic update operator docs from VictoriaMetrics/operator@ae6e1b6 (#5964) 2024-03-17 23:16:52 +02:00
Daria Karavaieva
dbdbbacf93
docs/vmanomaly: fix 404 links (#5968) 2024-03-17 23:16:15 +02:00
hagen1778
770dd7affd
docs: mention missing vmalert change for memory usage reduction in 1.97.3
521f9ffb43
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-17 23:15:23 +02:00
hagen1778
41a1efbea8
docs: follow-up 15e33d56f1
Update documentation according to changes in deduplication logic.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-17 23:07:51 +02:00
Andrii Chubatiuk
a58a81d80b
lib/streamaggr: pick sample with bigger timestamp or value on deduplicator (#5939)
Apply the same deduplication logic as in https://docs.victoriametrics.com/#deduplication
This would require more memory for deduplication, since we need to track timestamp
for each record. However, deduplication should become more consistent.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-03-17 23:06:37 +02:00
Github Actions
61dc9f5c8d
Automatic update operator docs from VictoriaMetrics/operator@de88675 (#5958) 2024-03-17 20:48:56 +02:00
Zakhar Bessarab
5f1f1f2982
docs: explicitly mention VMUI is available in cluster (#5955)
It is confusing for cluster users to find that VMUI is available at vmselect as it is only mentioned in the list of URLs. Explicit mention of vmselect URL in docs will make it easier to discover.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-03-17 20:48:07 +02:00
hagen1778
7b2f19a63e
docs: mention bug investigation in 1.99
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-17 20:46:48 +02:00
Hui Wang
349564fd82
vmalert: deprecate cmd-line flag -datasource.lookback (#5877)
* vmalert: deprecate cmd-line flag `-datasource.lookback`

* fix lint

* review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e80b44f19d)
2024-03-12 16:17:40 +01:00
Aliaksandr Valialkin
abf347ec0f
docs: replace speed up with more clear accelerate wording 2024-03-12 03:03:15 +02:00
Aliaksandr Valialkin
f92d4609e2
lib/storage: optimize /api/v1/labels and /api/v1/label/.../values when match[] contains metric name
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055
2024-03-12 03:01:47 +02:00
nemobis
14b913eb51
docs: fix typo in stalenes (#5950)
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-03-12 02:59:43 +02:00
Github Actions
221f0a000b
Automatic update operator docs from VictoriaMetrics/operator@9b1a6e6 (#5946) 2024-03-12 02:57:08 +02:00
Artem Navoiev
1348a822f8
docs:vmbackup fix typo sped -> speed
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-03-12 02:55:08 +02:00
Zakhar Bessarab
217893921c
docs: fix typo in flags description (#5942)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-03-12 02:45:46 +02:00
Github Actions
b2b034ddb7
Automatic update operator docs from VictoriaMetrics/operator@9caa896 (#5941) 2024-03-10 13:00:35 +02:00
Github Actions
9d57092a58
Automatic update operator docs from VictoriaMetrics/operator@403a78a (#5932) 2024-03-10 13:00:22 +02:00
hagen1778
86abbe5b67
app/vmctl: support TLS configuration for VictoriaMetrics destination
VictoriaMetrics destination is specified via `--vm-*` cmd-line flags
and is used in opentsdb, influx, prometheus, remote-read modes.

updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5426

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0b7ce70df4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-08 21:15:23 +01:00
hagen1778
b1fe25792c
app/vmctl: follow-up b9f7c3169a
* fix typos in flags description
* move the change to #tip section in changelog

b9f7c3169a
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 83a8c24281)
2024-03-08 21:15:09 +01:00
Khushi Jain
98c0909a12
app/vmctl : Provide TLS config options for vm native protocol (#5824)
Co-authored-by: Khushi Jain <khushi.jain@nokia.com>
(cherry picked from commit b9f7c3169a)
2024-03-08 21:15:09 +01:00
Aliaksandr Valialkin
b21fb355fa
docs/vmauth.md: update -help output after e08b91baafc95da090f75e9c29a27d8f62a2b76e 2024-03-07 01:37:48 +02:00
Aliaksandr Valialkin
b9788d5cf8
docs/CHANGELOG.md: typo fixes 2024-03-07 01:35:41 +02:00
Aliaksandr Valialkin
d9e3670627
lib/promauth: set the Host header to tlsServerName if itsn't empty
If tlsServerName isn't empty, then it is likely the https request is sent to IP instead of hostname.
In this case the request will fail, since Go automatically sets the Host header to the IP instead
of the desired hostname at tlsServerName. So set the Host header to tlsServerName if itsn't empty.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802
2024-03-07 01:23:40 +02:00
Aliaksandr Valialkin
e7e9721476
docs/vmauth.md: typo fixes after 7b2b980181 2024-03-07 01:08:40 +02:00
Aliaksandr Valialkin
93ef217857
app/vmauth: allow discovering backend ips behind shared hostname and spreading load among the discovered ips
This is done with the `discover_backend_ips` option at `user` and `url_map` level.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
2024-03-07 01:02:38 +02:00
Aliaksandr Valialkin
b2e182ca62
app/vmauth: add src_headers option at url_map, which allows routing incoming requests to different backends depending on request headers 2024-03-06 21:56:42 +02:00
Aliaksandr Valialkin
5bd4325135
app/vmauth: simplify configuration for src_query_args
Use the shorter form:

src_query_args:
- arg1=value1
- arg2=value2

instead of

src_query_args:
- name: arg1
  value: value2
- name: arg2
  value: value2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 21:24:53 +02:00
Aliaksandr Valialkin
1df7e5b69d
deployment/docker: update Go builder from Go1.21.7 to Go1.22.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved
2024-03-06 21:04:44 +02:00
Aliaksandr Valialkin
0929f255f0
docs/vmauth.md: mention that request query args can used for routing decisions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:58:10 +02:00
Aliaksandr Valialkin
87d59cdfa5
app/vmauth: add ability to route requests based on HTTP query args via src_query_args option
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:52:42 +02:00
Aliaksandr Valialkin
2711732944
app/vmselect/prometheus: do not drop match[] filters if -search.ignoreExtraFiltersAtLabelsAPI flag is set
The `match[]` filter is mandatory at /api/v1/series, so it mustn't be dropped here.

There is no sense in dropping `match[]` filter together with `extra_label` and `extra_filters[]`
at /api/v1/labels and /api/v1/label/.../values if -search.ignoreExtraFiltersAtLabelsAPI commnad-line flag is set,
since:
- the `match[]` filter triggers slow path at these APIs;
- the `extra_label` and `extra_filters[]` filters narrow down the number of matched time series,
  so they improve performance comparing to the case when only `match[]` filter is left,
  while `extra_label` and `extra_filters[]` filters are dropped.

This is a follow-up for 0b7a23a91d
2024-03-06 13:57:20 +02:00
Daria Karavaieva
98b56476d7
redirect alias (#5934) 2024-03-06 13:54:48 +02:00
Yury Molodov
56e61bb54b
vmui: improve tracing styles (#5926)
Improved trace display for better visual separation of branches:
* Increased left padding for each element
* Added padding for the last element in the branch
2024-03-06 13:54:07 +02:00
Github Actions
cfce708eb9
Automatic update operator docs from VictoriaMetrics/operator@f028fdf (#5929) 2024-03-06 13:34:37 +02:00
hagen1778
a4cdfaea28
docs: clarify deduplication is needed in multi-retention setup
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-06 13:33:40 +02:00
hagen1778
a364de3cfe
lib/writeconcurrencylimiter: mention dependency on CPU cores for -maxConcurrentInserts flag
The change also removes misleading `default` value from README for `maxConcurrentInserts`
cmd-line flag.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 73f5fb0f0c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-05 18:56:38 +01:00
Aliaksandr Valialkin
27b9e8ed3e
app/{vmagent,vminsert}: add -streamAggr.dropInputSamples command-line flag for dropping the specified labels from input samples before deduplication and streaming aggregation 2024-03-05 02:27:27 +02:00
hagen1778
b612c0536e
docs: re-fresh vmctl docs
* add recommendation about network bandiwdth between vmctl, source and destination
* use more relevant time series selector in examples - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5835

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-05 00:49:48 +02:00
Aliaksandr Valialkin
c38c45d71f
app/{vminsert,vmagent}: allow using -streamAggr.dedupInterval without -streamAggr.config
This allows performing online de-duplication of incoming samples
2024-03-05 00:47:23 +02:00
Github Actions
b530f6bf73
Automatic update operator docs from VictoriaMetrics/operator@c8ff654 (#5918) 2024-03-04 19:12:59 +02:00
Aliaksandr Valialkin
48a425898a
lib/streamaggr: enable time alignment for aggregate flushed to multiples of interval
For example, if `interval: 1m`, then data flush occurs at the end of every minute,
while `interval: 1h` leads to data flush at the end of every hour.

Add `no_align_flush_to_interval` option, which can be used for disabling the alignment.
2024-03-04 06:23:35 +02:00
Aliaksandr Valialkin
3ba9b2225e
docs/stream-aggregation.md: add troubleshooting section with solutions for common problems in streaming aggregation 2024-03-04 03:04:59 +02:00
Aliaksandr Valialkin
492c6c3ff5
docs/CHANGELOG.md: typo fix 2024-03-02 04:52:41 +02:00
Aliaksandr Valialkin
d0e6541f35
docs/stream-aggregation.md: typo fixes 2024-03-02 04:35:37 +02:00
Aliaksandr Valialkin
b912a45220
docs/stream-aggregation.md: remove superflouous output_relabel_configs from the config example for histogram aggregation 2024-03-02 03:36:08 +02:00
Aliaksandr Valialkin
0d5d46f9db
lib/streamaggr: huge pile of changes
- Reduce memory usage by up to 5x when de-duplicating samples across big number of time series.
- Reduce memory usage by up to 5x when aggregating across big number of output time series.
- Add lib/promutils.LabelsCompressor, which is going to be used by other VictoriaMetrics components
  for reducing memory usage for marshaled []prompbmarshal.Label.
- Add `dedup_interval` option at aggregation config, which allows setting individual
  deduplication intervals per each aggregation.
- Add `keep_metric_names` option at aggregation config, which allows keeping the original
  metric names in the output samples.
- Add `unique_samples` output, which counts the number of unique sample values.
- Add `increase_prometheus` and `total_prometheus` outputs, which ignore the first sample
  per each newly encountered time series.
- Use 64-bit hashes instead of marshaled labels as map keys when calculating `count_series` output.
  This makes obsolete https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5579
- Expose various metrics, which may help debugging stream aggregation:
  - vm_streamaggr_dedup_state_size_bytes - the size of data structures responsible for deduplication
  - vm_streamaggr_dedup_state_items_count - the number of items in the deduplication data structures
  - vm_streamaggr_labels_compressor_size_bytes - the size of labels compressor data structures
  - vm_streamaggr_labels_compressor_items_count - the number of entries in the labels compressor
  - vm_streamaggr_flush_duration_seconds - a histogram, which shows the duration of stream aggregation flushes
  - vm_streamaggr_dedup_flush_duration_seconds - a histogram, which shows the duration of deduplication flushes
  - vm_streamaggr_flush_timeouts_total - counter for timed out stream aggregation flushes,
    which took longer than the configured interval
  - vm_streamaggr_dedup_flush_timeouts_total - counter for timed out deduplication flushes,
    which took longer than the configured dedup_interval
- Actualize docs/stream-aggregation.md

The memory usage reduction increases CPU usage during stream aggregation by up to 30%.

This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5850
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5898
2024-03-02 03:15:43 +02:00
hagen1778
8ce65e93d0
docs: mention docs link for VictoriaLogs docker env
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-01 21:34:15 +02:00
Aliaksandr Valialkin
7559b6eb97
docs: update -help output after recent changes in VictoriaMetrics components 2024-03-01 05:31:13 +02:00
Aliaksandr Valialkin
0dbcc07d04
docs/CHANGELOG.md: typo fix 2024-03-01 04:41:39 +02:00
Aliaksandr Valialkin
5f0dd8743c
docs: bump the latest VictoriaMetrics release from v1.98.0 to v1.99.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0
2024-03-01 04:24:11 +02:00
Aliaksandr Valialkin
be1446f720
docs/LTS-releases.md: update latest LTS release to v1.97.3 and v1.93.13
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.3
and https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.13
2024-03-01 04:21:25 +02:00
Aliaksandr Valialkin
2540b8fdf8
deployment: update VictoriaLogs docker image from v0.4.2-victorialogs to v0.5.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs
2024-03-01 04:18:59 +02:00
Aliaksandr Valialkin
d66680fc47
docs/CHANGELOG.md: cut v1.99.0 release 2024-03-01 03:33:44 +02:00
Aliaksandr Valialkin
dc91c07384
docs/CHANGELOG.md: document v1.97.3 LTS release 2024-03-01 03:31:19 +02:00
Aliaksandr Valialkin
e4a1d6774e
docs: consistently use https://docs.victoriametrics.com/lts-releases/ link for LTS releases 2024-03-01 02:52:41 +02:00
Aliaksandr Valialkin
0fac8d6e87
docs/CHANGELOG.md: document v1.93.13 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.13
2024-03-01 02:41:05 +02:00
Aliaksandr Valialkin
bde2de3298
docs/VictoriaLogs/CHANGELOG.md: typo fix: bufix -> bugfix 2024-03-01 02:41:04 +02:00
Aliaksandr Valialkin
9a470239cf
docs/VictoriaLogs/CHANGELOG.md: cut v0.5.0-victorialogs 2024-03-01 01:43:35 +02:00
Aliaksandr Valialkin
d84cdf98da
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after e130f29659
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152
2024-03-01 00:51:49 +02:00
Yury Molodov
c71c2b37c5
vmui: add gap display option for charts #5152 (#5862) 2024-03-01 00:51:48 +02:00
Aliaksandr Valialkin
728fb59e0d
docs: update docs after 0b7a23a91d 2024-03-01 00:18:46 +02:00
Aliaksandr Valialkin
6aea1e5093
app/{vmagent,vminsert}: follow-up for 434a5803e7
Document the /opentelemetry/v1/metrics endpoint instead of /opentelemetry/api/v1/push,
since the /v1/metrics suffix is hardcoded in OpenTelemetry protocol specification.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5871
2024-02-29 18:05:12 +02:00
Nikolay
abb773120d
app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path (#5871)
* app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path
it must fix compatibility with opentemetry-collector [spec](https://opentelemetry.io/docs/specs/otlp/\#otlphttp-request)
this suffix is hard-coded and cannot be changed with collector configuration

* Apply suggestions from code review

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 17:59:57 +02:00
Github Actions
5e20be845f
Automatic update operator docs from VictoriaMetrics/operator@7e8f811 (#5902) 2024-02-29 17:53:11 +02:00
Alexander Marshalov
b8c588a7e2
fixed broken link in docs to google pub sub (#5901) 2024-02-29 17:11:46 +02:00
Hui Wang
25e454be4c
metricsql: fix label_join() when dst_label is equal to one of the `… (#5886)
* metricsql: fix label_join() when `dst_label` is equal to one of the `src_label`

* Update app/vmselect/promql/transform.go

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 16:02:48 +02:00
Aliaksandr Valialkin
cfeae6fffb
docs/VictoriaLogs/CHANGELOG.md: document the bugfix at a5795f533d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5897
2024-02-29 15:34:57 +02:00
Aliaksandr Valialkin
7310605c27
docs/CHANGELOG.md: clarify the description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5859
This is a follow-up for 3c74aa6b3d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5888
2024-02-29 15:28:52 +02:00