Commit Graph

9464 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
4837dc6e09
docs/VictoriaLogs/querying/vlogscli.md: replace vlogsql with vlogscli everywhere in the docs
`vlogsql` was the initial name of the command-line utility for interactive querying of logs.
Later it was renamed to `vlogscli`.

Thanks to @grinapo for the bugreport at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7512
2024-11-12 18:21:00 +01:00
Aliaksandr Valialkin
33a4f275b1
app/vmauth: properly inherit user-level options at url_map when url_prefix isnt set at the user level
The following user-level options must be unconditionally inherited by url_map, since this is what most users expect:

- retry_status_codes
- load_balancing_policy
- drop_src_path_prefix_parts
- discover_backend_ips

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7519
2024-11-12 17:53:19 +01:00
Hui Wang
32b89447ae
dashboards: add file label filter to vmalert dashboard panels (#7515)
Previously, metrics from groups with the same name but in different
files could be mixed in the results.

e.g. the evaluation time
[here](https://grafana.maas.victoriametrics.com/d/LzldHAVnz/victoriametrics-vmalert?orgId=1&var-ds=PE8D8DB4BEE4E4B22&var-job=All&var-instance=All&var-file=%2Fetc%2Fvmalert%2Fconfig%2Fvm-per-tenant-rulefiles-0%2Fmaas-tenant-1011-maas-1011-vm-health.yaml&var-group=All&var-topk=5&editPanel=23)
is the total for multiple groups from different tenants.
2024-11-12 09:00:39 -07:00
Github Actions
254d9e2729
Automatic update helm docs from VictoriaMetrics/helm-charts@ae2dabc (#7522)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-12 09:00:19 -07:00
Github Actions
8b0f5b2315
Automatic update operator docs from VictoriaMetrics/operator@593d523 (#7510)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: Amper <495795+Amper@users.noreply.github.com>
2024-11-12 08:59:51 -07:00
Zhu Jiekun
58ae3772fe
vmctl: add docs migrating from cluster to single with --vm-native-disable-binary-protocol (#7520)
### Describe Your Changes

when migrating from cluster to single, the native export API attach
`vm_account_id` and `vm_project_id` labels.
`--vm-native-disable-binary-protocol` is a workaround to remove them,
which is available since v1.93.0 but did not documented in vmctl doc.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4716

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-12 08:59:04 -07:00
Artem Fetishev
58dae07b7a
tests: cover key concepts with more tests (#7516)
More key concept tests

-  Verify how the time range points are calculated
-  Vefify that a range query is equivalent to many instant queries

Fix docs accordingly.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-11-12 08:57:54 -07:00
Aliaksandr Valialkin
83fc33af89
app/vmauth: simplify the logic for the fix at a0a154511a
The fix at a0a154511a looks too complicated and fragile:

- It moves buMin initialization to the place, which is far from its usage.
- It embeds unclear logic on selecting the proper buMin if it is broken,
  into unrelated loop.

The actual fix must be more clear:

$ git diff 95acca6b52 -- app/vmauth/

-               if n := bu.concurrentRequests.Load(); n < minRequests {
+               if n := bu.concurrentRequests.Load(); n < minRequests || buMin.isBroken() {

This should simplify further maintenance of this code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7489
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3061
2024-11-12 16:43:07 +01:00
Fred Navruzov
47b7487b5f
docs: vmanomaly - release 1.18.1 (#7517)
Some checks are pending
publish-docs / Build (push) Waiting to run
### Describe Your Changes

doc updates for vmanomaly v1.18.1

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-12 10:20:29 +02:00
Github Actions
7e72848ab3
Automatic update helm docs from VictoriaMetrics/helm-charts@27cf093 (#7498)
Some checks failed
publish-docs / Build (push) Has been cancelled
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-10 16:55:33 -07:00
Roman Khavronenko
6857736f04
docs: recommend using available_from in CONTRIBUTING and release gu… (#7493)
…ides

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-10 16:54:46 -07:00
Roman Khavronenko
564c309b11
docs: use available_from shortcode for versioning features (#7492)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-10 16:54:28 -07:00
Aliaksandr Valialkin
0f2233aef3
Add --delete option to rsync command, so it removes non-existing files from the destination
Some checks are pending
publish-docs / Build (push) Waiting to run
- The --delete option is needed to be passed to rsync during backups,
Since otherwise the backup may contain superfluous files after the second run of rsync,
because these files can be already removed at the source because of background merge

- the --delete option is needed when restoring from backup in order to remove superfluous files
from the destination directory. Otherwise these files may lead to inconsistent data at VictoriaLogs.
2024-11-10 10:10:29 +02:00
Aliaksandr Valialkin
cc908122bd
docs/VictoriaLogs: remove support for alerting from the roadmap, since it has been already implemented in the commit 68bad22fd2
Some checks failed
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
publish-docs / Build (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7255
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706
2024-11-09 00:09:21 +01:00
Aliaksandr Valialkin
bd106de2b2
deployment/docker: update VictoriaLogs Docker image from v0.41.0-victorialogs to v0.42.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.42.0-victorialogs
2024-11-08 23:51:56 +01:00
Aliaksandr Valialkin
23aa1897c6
deployment: update Go builder from Go1.23.1 to Go1.23.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.23.3+label%3ACherryPickApproved
2024-11-08 23:40:33 +01:00
Zhu Jiekun
4602752003
docs: [VictoriaLogs] FAQ add logs without msg field (#7487)
### Describe Your Changes

Add FAQs to VictoriaLogs:
- I want to ingest logs without message field, is that possible?
- What if my logs have multiple message fields candidates

Preview:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/docs/VL-FAQ-empty-msg/docs/VictoriaLogs/FAQ.md#i-want-to-ingest-logs-without-message-field-is-that-possible

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-08 23:32:56 +01:00
Aliaksandr Valialkin
0eb3a0a902
docs/VictoriaLogs/CHANGELOG.md: cut v0.42.0-victorialogs 2024-11-08 23:29:18 +01:00
Aliaksandr Valialkin
546bf7d579
lib/logstorage: properly skip filtered out dict values when calculating uniq_values, min, max, row_min and row_max stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-08 23:21:21 +01:00
Aliaksandr Valialkin
2f1ce74d97
lib/logstorage: properly clone field values at values stats function
Previously field values weren't properly cloned, which could lead to garbage output for `values` stats function

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-08 23:16:42 +01:00
Aliaksandr Valialkin
9b766d3e32
lib/logstorage: simplify the code for uniq_values stats function a bit
Move the repeated check for an empty value into statsUniqValuesProcessor.updateState() function.
This allow removing duplicate code for this check from statsUniqValuesProcessor.updateState() call sites.
2024-11-08 22:52:34 +01:00
Aliaksandr Valialkin
342f84c569
app/vlinsert/loki: show the original request body on parse errors
This should simplify debugging.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7490
2024-11-08 22:00:58 +01:00
Aliaksandr Valialkin
b7c290ea5e
docs/VictoriaLogs/CHANGELOG.md: refer to the issue related to adding fields to Syslog logs
This is a follow-up for cd60a4c589
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354
2024-11-08 21:50:14 +01:00
Aliaksandr Valialkin
4f0bec6f03
app/vlinsert/syslog: allow changing the default set of log fields to use as stream fields during syslog data ingestion
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7488
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7480

See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#stream-fields
2024-11-08 21:21:08 +01:00
Aliaksandr Valialkin
cd60a4c589
app/vlinsert/syslog: add an ability to drop and add fields during data ingestion via Syslog protocol
See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#dropping-fields
and https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#adding-extra-fields
2024-11-08 20:57:59 +01:00
Aliaksandr Valialkin
63c76b9b27
lib/logstorage: support for [label1=value1 ... labelN=valueN] syntax inside syslog messages for adding arbitrary labels (fields) to log entries 2024-11-08 19:57:22 +01:00
Aliaksandr Valialkin
f55791f20b
app/vlogscli: allow toggling wrapping long lines with \wrap_long_lines command
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
2024-11-08 17:08:50 +01:00
Roman Khavronenko
2febd00bb3
tests: couple applications and test suit (#7476)
* make test suite responisble for stopping apps
* reuse test suite fields to simplify function signatures

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit e60cce54a8)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-08 17:06:55 +01:00
Aliaksandr Valialkin
62e6c9bd6f
docs/VictoriaLogs/README.md: add Security chapter
It is inspired by https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428#discussion_r1831555716
2024-11-08 16:43:55 +01:00
Aliaksandr Valialkin
a44787372f
app/vlselect/logsql: add an ability to delay returning matching logs from live tailing via offset query arg
By default the delay equals to 1 second.

While at it, document refresh_interval query arg at /select/logsql/tail endpoint.

Thanks to @Fusl for the idea and the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428
2024-11-08 16:43:55 +01:00
Aliaksandr Valialkin
e5537bc64d
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters 2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
a98fb495c6
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
66b2987f49
lib/logstorage: optimize query imeediately after its parsing
This eliminates possible bugs related to forgotten Query.Optimize() calls.

This also allows removing optimize() function from pipe interface.

While at it, drop filterNoop inside filterAnd.
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
0550093802
app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint.
This reduces the probability of forgotten call to Query.Optimize().
2024-11-08 16:43:53 +01:00
Aliaksandr Valialkin
5a6531b329
lib/logstorage: add an ability to add prefix to resulting query field names in join pipe
See https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe
2024-11-08 16:43:53 +01:00
Andrii Chubatiuk
30dd4cdc0d
docs: add make command for docs version update (#7430)
### Describe Your Changes

added make target, which updates `{{% available_from "#" %}}` shortcode
to `{{% available_from "$(PKG_TAG)" %}}` if PKG_TAG matches expression
`v.*`. `{{% available_from %}}` shortcode was introduced in
https://github.com/VictoriaMetrics/vmdocs/pull/89 to show a reference to
a version in a changelog since which a feature was introduced

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7376

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-08 16:35:08 +01:00
nemobis
690328028d
docs: Update RELEX Oy figures (#7491)
Update figures for the existing RELEX Oy case study.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
2024-11-08 07:13:05 -08:00
hagen1778
8e0fbb0ed2
docs: test available_from shortcode for versioning
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-08 16:06:27 +01:00
hagen1778
acb9f47e8d
docs: fix OpenTSDB port for cluster version in API examples
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-08 12:59:30 +01:00
Roman Khavronenko
a0a154511a
app/vmauth: properly check for backend health before leastLoaded policy (#7489)
Previously, vmauth could have pick `buMin` as least loaded backend
without checking its status. In result, vmauth could have respond to the
user with an error even if there were healthy backends. That could
happen if healthy backends already had non-zero amount of concurrent
requests executing at the moment of least-loaded backend choosing logic.

Steps to reproduce:
1. Setup vmauth with two backends: healthy and non-healthy
2. Execute a bunch of concurrent requests against vmauth (i.e. Grafana
dash reload)
3. Observe that some requests will fail with message that all backends
are unavailable

Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3061
---
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-08 11:45:16 +01:00
Andrii Chubatiuk
b399f1c656
victorialogs: added missing changelog entry about DataDog support (#7464)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
### Describe Your Changes

Added missing changelog entry for PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5536

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-07 14:26:02 +01:00
Github Actions
af9638c480
Automatic update helm docs from VictoriaMetrics/helm-charts@1d4f6d6 (#7473)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-07 14:24:36 +01:00
Github Actions
156a04e900
Automatic update operator docs from VictoriaMetrics/operator@ee35f90 (#7474)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-11-07 14:24:02 +01:00
Github Actions
39fee4fd98
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@0b86b62 (#7477) 2024-11-07 14:23:42 +01:00
Github Actions
eb4ace8741
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@aad9171 (#7478) 2024-11-07 14:23:28 +01:00
Andrii Chubatiuk
d96678a0c3
dashboards/victorialogs: allow multiple values for instance variable (#7465)
### Describe Your Changes

Related issue https://github.com/VictoriaMetrics/helm-charts/issues/1699

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-07 14:22:53 +01:00
hagen1778
f553628f46
tests: fix minor typos
* rename vmsingle test to actually match the mask
* swap got/want arguments, as they were misplaced

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-07 13:27:20 +01:00
Artem Fetishev
7c60d82215
tests: integration tests for vmsingle (#7434)
### Describe Your Changes

This PR continues the implementation of integration tests (#7199). It
adds the support for vm-single:

-    A vmsingle app wrapper has been added
- Sample vmsingle tests that test the VM documentation related to
querying data (#7435)
- The tests use the go-cmp/{cmp,/cmpopts} packages, therefore they have
been added to ./vendor
-    Minor refactoring: data objects have been moved to model.go

Advice on porting things to cluster branch:

- The build rule must include tests that start with TestVmsingle
(similarly to how TestCluster tests are skipped in master branch)
- The build rule must depend on `vmstorage vminsert vmselect` instead of
`victoria-metrics`
- The query_test.go can actually be implemented for cluster as well. To
do this the tests need to be renamed to start with TestCluster and the
tests must instantiace vm{storage,insert,select} instead of vmsingle.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-11-07 12:58:37 +01:00
Phuong Le
4d383fdb9a
deployment/docker/base: fix the typo that causes InvalidDefaultArgInFrom 2024-11-07 11:33:03 +01:00
Aliaksandr Valialkin
d1af84a49d
docs/VictoriaLogs/LogsQL.md: add missing reference to pipes docs from join pipe docs
Some checks failed
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
CodeQL JS/TS / Analyze (push) Has been cancelled
This is needed for consistency with other pipe docs
2024-11-06 20:59:25 +01:00