Commit Graph

4225 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
252ff71a49
docs/VictoriaLogs/data-ingestion/README.md: document that Rsyslog and Syslog-ng can be used for ingesting logs into VictoriaLogs 2024-06-25 15:15:53 +02:00
Aliaksandr Valialkin
0a88a55ae0
docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release 2024-06-25 14:59:31 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
hagen1778
26d58f8768
docs: add missing ticket links to the recent releases
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 14:54:24 +02:00
hagen1778
841305da8a
deployment: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0dcb3acbf7)
2024-06-25 13:58:59 +02:00
Nikolay
bf1464fc33
dashboards: add dashboard and alerts for vmauth (#6491)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 14b9ef1e4d)
2024-06-25 11:17:04 +02:00
hagen1778
8237b63fe3
docs: make docs-sync
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cc6a1da0d7)
2024-06-25 11:17:04 +02:00
Roman Khavronenko
e3da10a7c1
docs: add section for docs modification (#6528)
While there, remove the obsolete files from the previous version of
docs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 87edfa61c1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 11:17:03 +02:00
Aliaksandr Valialkin
a5445e09c2
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 5ecf439078.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin
813a7aec6c
docs: return back spellcheck fixes from the commit 92b22581e6, which were accidentally reverted in 45cf83cd8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470
2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow (#6470)"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin
7026498359
lib/httpserver: revert 9b7e532172
Reason for revert: this commit doesn't resolve real security issues,
while it complicates the resulting code in subtle ways (aka security circus).

Comparison of two strings (passwords, auth keys) takes a few nanoseconds.
This comparison is performed in non-trivial http handler, which takes thousands
of nanoseconds, and the request handler timing is non-deterministic because of Go runtime,
Go GC and other concurrently executed goroutines. The request handler timing is even
more non-deterministic when the application is executed in shared environments
such as Kubernetes, where many other applications may run on the same host and use
shared resources of this host (CPU, RAM bandwidth, network bandwidth).

Additionally, it is expected that the passwords and auth keys are passed via TLS-encrypted connections.
Establishing TLS connections takes additional non-trivial time (millions of nanoseconds),
which depends on many factors such as network latency, network congestion, etc.

This makes impossible to conduct timing attack on passwords and auth keys in VictoriaMetrics components.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6423/files
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392
2024-06-25 01:51:06 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
8dd69fd7ae
docs/CHANGELOG.md: cut v1.102.0-rc2 release 2024-06-24 23:22:45 +02:00
Nikolay
7ab046cf88
vendor update: updates metrics to v1.34.0 (#6523)
mention change for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1af13208c2)
2024-06-24 14:52:32 +02:00
Ivan Yatskevich
50343768fd
Publish Alertmanager configuration docs for Managed VictoriaMetrics (#6428)
### Describe Your Changes

With the recent release of Managed VictoriaMetrics users are able to
create and execute Alerting & Recording rules and send notifications via
hosted Alertmanager.

So, we're publishing Alertmanager configuration docs for Managed
VictoriaMetrics.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1106d91741)
2024-06-24 12:50:15 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow (#6470)
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Andrii Chubatiuk
50783fca4d
app/vmagent: add max_scrape_size to scrape config (#6434)
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1e83598be3)
2024-06-20 14:00:22 +02:00
Hui Wang
a393b993d6
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512

(cherry picked from commit bc37b279aa)
2024-06-20 14:00:19 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
Zakhar Bessarab
886f545f81
lib/fs/fscore: do not trim content from path (#6503)
### Describe Your Changes

Trimming content which is loaded from an external pass leads to obscure
issues in case user-defined input contained trimmed chars. For example.
user-defined password "foo\n" will become "foo" while user will expect
it to contain a new line.

---
For example, a user defines a password which ends with `\n`. This often
happens when user Kubernetes secrets and manually encodes value as
base64-encoded string.

In this case vmauth configuration might look like:
```
users:
  - url_prefix:
      - http://vminsert:8480/insert/0/prometheus/api/v1/write
    name: foo
    username: foo
    password: "foobar\n"
```

vmagent configuration for this setup will use the following flags:
```
-remoteWrite.url=http://vmauth:8427/
-remoteWrite.basicAuth.passwordFile=/tmp/vmagent-password
-remoteWrite.basicAuth.username="foo"
```
Where `/tmp/vmagent-password` is a file with `foobar\n` password.

Before this change such configuration will result in `401 Unauthorized`
response received by vmagent since after file content will become
`foobar`.

---
An example with Kubernetes operator which uses a secret to reference the
same password in multiple configurations.

<details>
  <summary>See full manifests</summary>

`Secret`:
```
apiVersion: v1
data:
  name: Zm9v # foo
  password: Zm9vYmFy # foobar\n
  username: Zm9v= # foo
kind: Secret
metadata:
  name: vmuser
```

`VMUser`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
  name: vmagents
spec:
  generatePassword: false
  name: vmagents
  targetRefs:
  - crd:
      kind: VMAgent
      name: some-other-agent
      namespace: example
  username: foo
  # note - the secret above is referenced to provide password
  passwordRef:
    name: vmagent
    key: password
```

`VMAgent`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: example
spec:
  selectAllByDefault: true
  scrapeInterval: 5s
  replicaCount: 1
  remoteWrite:
    - url: "http://vmauth-vmauth-example:8427/api/v1/write"
      # note - the secret above is referenced as well
      basicAuth:
        username:
          name: vmagent
          key: username
        password:
          name: vmagent
          key: password
```

</details>

Since both config target exactly the same `Secret` object it is expected
to work, but apparently the result will be `401 Unauthrized` error.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 201fd6de1e)
2024-06-19 10:37:12 +02:00
Nihal
8fd46caa22
victoria-metrics: constant-time comparison of credentials like authkeys and basic auth credentials (#6423)
Changes for constant-time comparison of credentials like authkeys and
basic auth credentials.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392

---------

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
(cherry picked from commit 9b7e532172)
2024-06-19 10:37:09 +02:00
Hui Wang
36f6a08f07
doc: add missing cmd flag description (#6504)
(cherry picked from commit 2259e3cac7)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 15:29:11 +02:00
Yury Molodov
88650abf97
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.

(cherry picked from commit 32fbffedd9)
2024-06-18 15:28:56 +02:00
Hui Wang
5be2f2c4e4
vmalert-tool: support file path with hierarchical patterns and regexp… (#6501)
…es, and http url in unittest cmd-line flag `-files`

(cherry picked from commit 3b8970802e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:17:42 +02:00
Hui Wang
b09da16899
vmalert-tool: exit normally when no rule is defined under rule group (#6502)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit d62f303e53)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:17:23 +02:00
Aliaksandr Valialkin
8650809435
docs/keyConcepts.md: document optional timeout query arg at /api/v1/query and /api/v1/query_range
This is a follow-up for ac9c2a796f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6020
2024-06-18 12:32:04 +02:00
Aliaksandr Valialkin
0ee360b48c
docs/VictoriaLogs/CHANGELOG.md: typo fix in the release date for v0.20.2-victorialogs 2024-06-18 00:28:49 +02:00
Aliaksandr Valialkin
ee114ca59c
app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint
This has been broken in 2b6a634ec0
2024-06-18 00:24:11 +02:00
Aliaksandr Valialkin
7658cc36a6
docs/VictoriaLogs/CHANGELOG.md: cut v0.20.1-victorialogs 2024-06-17 23:29:28 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
f74e6b0674
app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425 2024-06-17 22:30:28 +02:00
jackyin
4a6bf7f218
app/vmui: copy button shows undefined (#6495)
### Describe Your Changes
fix #6421
some aggregation func don't return \_\_name\_\_ value
2024-06-17 22:30:28 +02:00
Roman Khavronenko
df7e300071
app/vmselect/promql: check for ranged vectors in aggr funcs if implicit conversions are disabled (#6450)
Check for ranged vector arguments in aggregate expressions when
`-search.disableImplicitConversion` or `-search.logImplicitConversion`
are enabled.
 For example, `sum(up[5m])` will fail to execute if these flags are set.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [*] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6149adbe10)
2024-06-17 14:25:43 +02:00
Aliaksandr Valialkin
938862946c
docs/VictoriaLogs: mention which collectors support sending logs to VictoriaLogs in syslog format 2024-06-17 13:13:51 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
Zhu Jiekun
9c3dabccac
docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494)
### Describe Your Changes
Add CHANGELOG for Statsd support in v1.102.0-rc1
- CHANGELOG is missing for #5053.
- It would be better to include it so that users can be aware of when
this feature is released, rather than attempting to use it with lower
version VM components. See discussion
[here](https://victoriametrics.slack.com/archives/CGZF1H6L9/p1718462859065049?thread_ts=1718451117.668789&cid=CGZF1H6L9).

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
Fred Navruzov
73fbce30bd
docs/vmanomaly - add config example for min_dev arg (#6490)
### Describe Your Changes

Added config example for `min_dev_from_expected` arg; also, small
styling fixes and alignments

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
hagen1778
43c53271bf
docs: update wording for 6e395048d3
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit da4fbf61a4)
2024-06-14 16:31:42 +02:00
Hui Wang
5ecf439078
app/vmselect: fix the way of counting raw samples in single query (#6464)
The limit is specified with command-line flag
`-search.maxSamplesPerQuery`.
Previously, samples might be over-counted and query can't be fixed by
reducing time range.
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

(cherry picked from commit 6e395048d3)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 16:24:31 +02:00
Andrii Chubatiuk
8ca1813bd2
lib/flagutil: use month limit for duration flag for parsed duration assessment (#6486)
use maxMonths limit for parsed duration flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit faf67aa8b5)
2024-06-14 15:21:32 +02:00
jackyin
f69495cd5f
app/vmalert: fix VMAlert oauth2 error (#6478)
Properly set ClientSecret param for notifier.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5223981fed)
2024-06-14 15:21:30 +02:00
hagen1778
64df1383e7
app/vmalert: properly configure authentication with S3 when -s3.configFilePath is specified.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 14:22:35 +02:00
Andrii Chubatiuk
abc233a902
lib/backup/s3remote: fixed credsFilePath flag (#6488)
properly use credsFilePath flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e678a9aa51)
2024-06-14 14:14:58 +02:00
Zakhar Bessarab
a91a2b97ed
docs/guides/otel-guide: update sentence to make it less obscure (#6483)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit f11261dc44)
2024-06-14 14:14:56 +02:00
Andrii Chubatiuk
779436bd9c
app/vmalert: fixed path prefixes for system routes (#6435)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit eea361defb)
2024-06-14 14:14:54 +02:00
Artem Navoiev
6c587eefbf
docs: add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit d1baa41b6a)
2024-06-14 13:26:47 +02:00
Fred Navruzov
455d761de9
docs/vmanomaly: fix-typos-in-v1.13-preset (#6484)
### Describe Your Changes

small fix of typos in v1.13 presets (vmanomaly docs)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1cf28598c6)
2024-06-14 13:26:44 +02:00
Roman Khavronenko
5df50e5645
lib/streamaggr: prevent rate_sum and rate_avg from producing NaNs (#6482)
### Describe Your Changes

* check if `lastValue` was seen at least twice with different
timestamps. Otherwise, the difference between last timestamp and
previous timestamp could be `0` and will result into `NaN` calculation
* check if there items left in lastValue map after staleness cleanup.
Otherwise, `rate_avg` could have produce `NaN` result.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 51d19485bb)
2024-06-14 13:26:42 +02:00
Github Actions
5683610bf8
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@bc562ef (#6481)
(cherry picked from commit 2d575cd27a)
2024-06-14 13:26:40 +02:00
Fred Navruzov
aa85b772bb
docs/vmanomaly: update preset guide and cross-links (#6480)
### Describe Your Changes

Updates and cross-references for `preset` mode of `vmanomaly` [in
docs](https://docs.victoriametrics.com/anomaly-detection/presets/)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2184c0c730)
2024-06-14 13:26:37 +02:00
Daria Karavaieva
ae39b02496
docs/vmanomaly: preset guide (#6151)
- added instructions of how to run presets
- description of Node Exporter indicators covered

(cherry picked from commit c32b8d39dd)
2024-06-14 13:26:29 +02:00
Aliaksandr Valialkin
b455d356fc
docs/CONTRIBUTING.md: add checks about backwards compatibility and historical conventions 2024-06-13 17:08:54 +02:00
Aliaksandr Valialkin
7af2b7905f
docs/CONTRIBUTING.md: various typo fixes 2024-06-13 17:02:05 +02:00
Zakhar Bessarab
ac16d1dc1b
lib/promscrape: increase default value for promscrape.maxDroppedTargets to 10_000 (#6459)
### Describe Your Changes
This limit can be increased since after
4513893ead
tracking of dropped targets uses much less memory per entry.

See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381#issuecomment-2156708228

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit 34071ac660)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-13 09:28:16 +02:00
LHHDZ
41e4135371
app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401)
This change fixes the following panic:
```
2024-06-04T11:16:52.899Z        warn    app/vmauth/auth_config.go:353   cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally
panic: runtime error: integer divide by zero

goroutine 9 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58
panic({0x103115100?, 0x10338d700?})
        /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124
main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210
main.(*URLPrefix).getBackendURL(0x140000aa080)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8
```

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-06-12 11:47:44 +02:00
Aliaksandr Valialkin
27efb9499b
docs/VictoriaLogs/logsql-examples.md: added an example on how to search for a phrase with whitespace 2024-06-11 19:13:47 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress 2024-06-11 17:51:01 +02:00
Fred Navruzov
ea90d91a30
docs/vmanomaly - fix 404 img links & remove outdated models (#6460)
### Describe Your Changes

Fix 404 relative img links for v1.13.0 update of vmanomaly docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 9a355c3512)
2024-06-11 17:05:10 +02:00
Fred Navruzov
0219d34b21
docs/vmanomaly - release 1.13.0 preparation (#6436)
### Describe Your Changes

[vmanomaly docs](https://docs.victoriametrics.com/anomaly-detection/)
update for changes, introduced in v1.13.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1feb5d04d7)
2024-06-11 17:05:07 +02:00
Nikolay
ae76794a19
follow-up docs update after 77f22fdb8d (#6454)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 33d07e915f)
2024-06-11 17:05:05 +02:00
Github Actions
c96b334b3e
Automatic update operator docs from VictoriaMetrics/operator@310a860 (#6449)
(cherry picked from commit 020d3f3d88)
2024-06-11 17:05:01 +02:00
Lapo Luchini
dd311aa76f
Small fix in MetricsQL.md (#6452)
### Describe Your Changes

value→valid

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Lapo Luchini <lapo@lapo.it>
(cherry picked from commit 14f586238f)
2024-06-11 17:04:59 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
Yury Molodov
2300e30ff3
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:39:25 +02:00
hagen1778
fdf0a936f0
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8d95522529)
2024-06-10 14:05:58 +02:00
Nikolay
0ce7f38e1c
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d44058bcd6)
2024-06-10 12:41:51 +02:00
Dmytro Kozlov
a4bdc14bc5
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 8b46bb0c41)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:44 +02:00
Hui Wang
028a80613f
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329,
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 61dce6f2a1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:29 +02:00
Andrii Chubatiuk
6fd314d8ba
vmagent: updated dashboard and alert for stream aggregation (#6427)
### Describe Your Changes

Added streaming aggregation section to vmagent dashboards
Added alert for streaming aggregation and deduplication flush timeouts
Removed deprecated compose versions from compose files

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2da45a8368)
2024-06-10 12:37:22 +02:00
Artem Navoiev
d6b56a1460
docs victorialogs: add alias for quickstart for VL to be consistent with other quickstarts
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 318e9e9de0)
2024-06-10 12:37:19 +02:00
Artem Navoiev
a383e00e48
add alias for vmalert-tools
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 8f254232bb)
2024-06-10 12:37:16 +02:00
Aliaksandr Valialkin
792dd697f4
docs/CHANGELOG.md: document v1.93.15 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.15
2024-06-07 23:43:20 +02:00
Aliaksandr Valialkin
f61bf790a4
docs/LTS-releases.md: update v1.97.4 LTS release to v1.97.5
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:16:55 +02:00
Aliaksandr Valialkin
37ca3a2b0c
docs/CHANGELOG.md: add changelog for v1.97.5 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:03:35 +02:00
Aliaksandr Valialkin
3f883559e2
docs/CHANGELOG.md: cut v1.202.0-rc1 release 2024-06-07 16:53:48 +02:00
Aliaksandr Valialkin
a38759f66a
docs: run make docs-sync after c740a8042e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362
2024-06-07 16:45:53 +02:00
Aliaksandr Valialkin
32aa0751a1
lib/streamaggr: follow-up for 7cb894a777
- Use bytesutil.InternString() instead of strings.Clone() for inputKey and outputKey in aggregatorpushSamples().
  This should reduce string allocation rate, since strings can be re-used between aggrState flushes.
- Reduce memory allocations at dedupAggrShard by storing dedupAggrSample by value in the active series map.
- Remove duplicate call to bytesutil.InternBytes() at Deduplicator, since it is already called inside dedupAggr.pushSamples().
- Add missing string interning at rateAggrState.pushSamples().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402
2024-06-07 16:35:53 +02:00
Roman Khavronenko
78121642df
lib/streamaggr: reduce number of inuse objects (#6402)
The main change is getting rid of interning of sample key. It was
discovered that for cases with many unique time series aggregated by
vmagent interned keys could grow up to hundreds of millions of objects.
This has negative impact on the following aspects:
1. It slows down garbage collection cycles, as GC has to scan all inuse
objects periodically. The higher is the number of inuse objects, the
longer it takes/the more CPU it takes.
2. It slows down the hot path of samples aggregation where each key
needs to be looked up in the map first.

The change makes code more fragile, but suppose to provide performance
optimization for heavy-loaded vmagents with stream aggregation enabled.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-06-07 16:35:52 +02:00
Roman Khavronenko
fae589bb83
lib/promrelabel: speedup label match by __name__ (#6432)
The change adds a fastpath for `equalValue` comparisons against
`__name__` label by avoiding calls to `toCanonicalLabelName` func. This
speedups matches by metric name like `'foo'`. See bench stats below:
```
benchcmp old.txt new.txt

benchmark                                           old ns/op     new ns/op     delta
BenchmarkIfExpression/equal_label:_last-10          35.6          35.1          -1.18%
BenchmarkIfExpression/equal_label:_middle-10        18.3          17.3          -5.41%
BenchmarkIfExpression/equal_label:_first-10         1.20          1.24          +2.74%
BenchmarkIfExpression/equal___name__:_last-10       10.1          4.96          -50.75%
BenchmarkIfExpression/equal___name__:_middle-10     5.79          3.16          -45.41%
BenchmarkIfExpression/equal___name__:_first-10      1.17          1.05          -9.76%
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-07 16:35:52 +02:00
Andrii Chubatiuk
93cd08f15f
lib/streamaggr: metrics to track dropped, nan samples and samples lag (#6358)
### Describe Your Changes

Added streamaggr metrics to:
 - `vm_streamaggr_samples_lag_seconds` - samples lag
- `vm_streamaggr_ignored_samples_total{reason="nan"}` - ignored NaN
samples
- `vm_streamaggr_ignored_samples_total{reason="too_old"}` - ignored old
samples

(cherry picked from commit 185fac03b3)
2024-06-06 19:22:45 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress 2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
6d986f731b
docs/VictoriaLogs/CHANGELOG.md: document the a68c2c0f17 2024-06-06 12:21:47 +02:00
Github Actions
2e3c039113
Automatic update operator docs from VictoriaMetrics/operator@32f7674 (#6414) 2024-06-06 12:12:58 +02:00
Aliaksandr Valialkin
c6b6900152
docs/VictoriaLogs/LogsQL.md: typo fix: replace from _json with from _msg where appropriate 2024-06-05 10:15:56 +02:00
Aliaksandr Valialkin
be4aefd133
docs/VictoriaLogs/LogsQL.md: substitute TAG with APP-NAME[PROCID] in rfc3164 syslog message format at unpack_syslog pipe docs 2024-06-05 10:13:58 +02:00
Aliaksandr Valialkin
a200fb433a
lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin
f8336783c8
docs/VictoriaLogs/LogsQL.md: typo fix: substitute "'" with "" in front of John` 2024-06-05 09:51:29 +02:00
Aliaksandr Valialkin
9e79a632c1
docs/VictoriaLogs/LogsQL.md: typo fix: its -> it 2024-06-05 09:46:35 +02:00
Zhu Jiekun
024ab786ce
docs: [deployment] update CHANGELOG.md to include go version change 1.22.4 (#6412)
### Describe Your Changes
Update CHANGELOG to include go version change introduced in
43cf221681

Also see:
- https://go.dev/doc/devel/release
-
https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-05 09:46:34 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
hagen1778
607007add9
docs: add technical articles by @jiekun
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-05 03:16:41 +02:00
pludov
2efd97a63c
lib/fs: support NFS implementations that return EEXIST instead of ENOTEMPTY (#6398)
### Describe Your Changes

Fix for issue #6396: according to rmdir manpage, ENOTEMPTY and EEXIST
should be treated equally

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Ludovic Pollet <ludovic.pollet@exfo.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3ddae77c63)
2024-06-04 15:30:48 +02:00
Aliaksandr Valialkin
1a1ee87b46
docs/VictoriaLogs/LogsQL.md: mention unpack_syslog pipe in the Transformations section 2024-06-04 03:17:14 +02:00
Aliaksandr Valialkin
7d31574071
docs/VictoriaLogs/Roadmap.md: remove LogsQL transformations from roadmap, since they are already implemented 2024-06-04 02:21:23 +02:00
Aliaksandr Valialkin
39d238592c
docs/VictoriaLogs/logsql-examples.md: typo fix 2024-06-04 02:17:59 +02:00
Aliaksandr Valialkin
53928a44d1
docs/VictoriaLogs/logsql-examples.md: typo fix 2024-06-04 02:10:56 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00