Commit Graph

9510 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
0f6b9e9490
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886
2024-10-30 23:20:36 +01:00
Aliaksandr Valialkin
c5d08d317c
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Aliaksandr Valialkin
2e635a42d8
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Dmytro Kozlov
6a6d08d03d
deployment/docker: update datasource versions to the latest releases (#7396)
### Describe Your Changes

Updated the versions of the data sources to the latest releases

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-31 01:18:38 +04:00
Aliaksandr Valialkin
f62502a943
docs/VictoriaLogs/querying/README.md: typo fix after f2cd284cf4a7429848fb4be1522483a9b93a43e7: Alls -> All
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
2024-10-30 19:08:57 +01:00
Aliaksandr Valialkin
7603446850
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365
2024-10-30 17:59:49 +01:00
Aliaksandr Valialkin
d2dce13df6
app/vlinsert: typo fix after 16ee470da6 2024-10-30 17:59:49 +01:00
Artem Fetishev
683f8c2780
dashboards: add Restarts panel (#7394)
Reopening PR #7373 from a branch in VictoriaMetrics repo in order to
enable edits and rebase.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 16:44:08 +01:00
hagen1778
6494606924
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 16:06:57 +01:00
Artem Fetishev
e614367cbf
tests: Initial version of integration tests (#7253)
### Describe Your Changes

Related issue: #7199

This is the initial version of the integration tests for cluster. See
`README.md` for details.

Currently cluster only, but it can also be used for vm-single if needed.

The code has been added to the apptest package that resides in the root
directory of the VM codebase. This is done to exclude the integration
tests from regular testing build targets because:

- Most of the test variants do not apply to integration testing (such as
pure or race).
- The integtation tests may also be slow because each test must wait for
2 seconds so vmstorage flushes pending content). It may be okay when
there are a few tests but when there is a 100 of them running tests will
require much more time which will affect the developer wait time and CI
workflows.
- Finally, the integration tests may be flaky especially short term.

An alternative approach would be placing apptest under app package and
exclude apptest from packages under test, but that is not trivial.

The integration tests rely on retrieving some application runtime info
from the application logs, namely the application's host:port. Therefore
some changes to lib/httpserver/httpserver.go were necessary, such as
reporting the effective host:port instead the one from the flag.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit d7b3589dbd)
2024-10-30 15:22:22 +01:00
Aliaksandr Valialkin
25bca94d04
deployment/docker: update VictoriaLogs from v0.38.0-victorialogs to v0.39.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.39.0-victorialogs
2024-10-30 15:11:17 +01:00
Aliaksandr Valialkin
13822f6d11
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release 2024-10-30 15:06:00 +01:00
Aliaksandr Valialkin
16ee470da6
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.
2024-10-30 14:59:38 +01:00
Aliaksandr Valialkin
96466562b6
docs/VictoriaLogs/data-ingestion/README.md: formatting fixes after 8b36529b32
- Remove leading whitespace from the first lines in 'HTTP parameters' chapter.
  This whitespace isn't needed for the markdown formatting.

- Add leading whitespace for the second sentence in the list bullet describing AccountID and ProjectID HTTP headers.
  This fixes markdown formatting for this list bullet.
2024-10-30 14:23:54 +01:00
Aliaksandr Valialkin
ed73f8350b
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.
2024-10-30 14:17:33 +01:00
Aliaksandr Valialkin
102e9d4f4e
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.
2024-10-30 13:39:28 +01:00
Dan Dascalescu
258ee93fd1
docs: clarify "single" in Single-server-VictoriaMetrics.md (#7369)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
### Describe Your Changes

"Single version" is unclear, since VM is also a single-executable. I
think "single-node" is clearer.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-30 12:55:52 +01:00
danish-mehmood
8dc4e2b5a5
docs: fix typos and format in case study (#7374)
### Describe Your Changes

- made small typo fix in case studies

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-30 12:48:09 +01:00
cangqiaoyuzhuo
45896fb477
chore: fix function name (#7381)
### Describe Your Changes

 fix function name

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-30 12:45:20 +01:00
hagen1778
cfba770c8e
docs: consistently update twitter.com to x.com
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7383

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:44:50 +01:00
Lubov66
702ff923fe
docs: replace twitter.com with x.com (#7383)
Replacing the name and link of the social network
2024-10-30 12:41:14 +01:00
Github Actions
b2cf8685e5
Automatic update helm docs from VictoriaMetrics/helm-charts@87ea94d (#7385)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-10-30 12:39:35 +01:00
hagen1778
2c93353173
docs: add frequently asked questions to vmalert integration with vlogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:38:47 +01:00
hagen1778
b789a9dc83
docs: mark alerting feature as done in VictoriaLogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:20:00 +01:00
hagen1778
77b690ab26
docs: update version placeholder to point to the next version
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:19:44 +01:00
Artem Navoiev
3473440d26
docs: logs vmalert add identifier
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-10-30 09:45:28 +01:00
hagen1778
2ac07aa813
docs: rm unused vmanomaly assets
Some checks failed
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
CodeQL JS/TS / Analyze (push) Has been cancelled
These images copies are present in vmanomaly subfolder.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:52:25 +01:00
hagen1778
a49eb7d4f5
docs: add step for updating VM version in docs
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:50:40 +01:00
hagen1778
b86b0dd910
docs: update VM versions to the latest version
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:48:36 +01:00
Aliaksandr Valialkin
c963d7d10d
docs/VictoriaLogs/CHANGELOG.md: remove unnededed with prefix in front of rank at top pipe example 2024-10-29 18:27:40 +01:00
Aliaksandr Valialkin
c43a6ce0eb
deployment/docker: update VictoriaLogs from v0.37.0-victorialogs to v0.38.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.38.0-victorialogs
2024-10-29 18:16:16 +01:00
Aliaksandr Valialkin
12223cf5d0
docs/VictoriaLogs/CHANGELOG.md: cut v0.38.0 release 2024-10-29 18:08:36 +01:00
Aliaksandr Valialkin
4f057e5669
app/vlselect/vmui: run make vmui-logs-update after dd89745a34
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7294
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7279
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7290
2024-10-29 18:06:51 +01:00
Aliaksandr Valialkin
7a623c225f
lib/logstorage: follow-up for af831a6c906158f371f1b6810706fa0a54b78386
Sync the code between top and sort pipes regarding the code related to rank.
2024-10-29 16:44:46 +01:00
Aliaksandr Valialkin
8faee6b446
app/vlogscli: print hint on how to see available commands when starting vlogscli
This should improve the first-time experience with vlogscli
2024-10-29 16:44:46 +01:00
Aliaksandr Valialkin
3c06d083ea
lib/logstorage: add an ability to return rank from top pipe results 2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
7a62eefa34
lib/logstorage: dynamically adjust the number of (bloom, values) shards in a part depending on the number of non-const columns
This allows reducing the amounts of data, which must be read during queries over logs with big number of fields (aka "wide events").
This, in turn, improves query performance when the data, which needs to be scanned during the query, doesn't fit OS page cache.
2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
67b4059aa4
docs/VictoriaLogs/README.md: add tuning chapter 2024-10-29 16:44:44 +01:00
Aliaksandr Valialkin
8d968acd0a
lib/logstorage: avoid reading columnsHeader data when field_values pipe is applied directly to log filters
This improves performance of `field_values` pipe when it is applied to large number of data blocks.
This also improves performance of /select/logsql/field_values HTTP API.
2024-10-29 16:44:44 +01:00
Hui Wang
68bad22fd2
vmalert: integrate with victorialogs (#7255)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.

Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.

Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.

Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 16:30:39 +01:00
Fred Navruzov
5d73b8b866
docs/vmanomaly - release 1.18.0 (#7378)
Some checks are pending
publish-docs / Build (push) Waiting to run
### Describe Your Changes

docs/vmanomaly - release 1.18.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-28 17:25:24 +02:00
Zakhar Bessarab
4e50d6eed3
lib/storage/partition: prevent panic in case resulting in-memory part is empty after merge (#7329)
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
It is possible for in-memory part to be empty if ingested samples are
removed by retention filters. In this case, data will not be discarded
due to retention before creating in memory part. After in-memory parts
merge samples will be removed resulting in creating completely empty
part at destination.

 This commit checks for resulting part and skips it, if it's empty.

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-27 20:40:13 +01:00
Zhu Jiekun
f06c7e99fe
lib/promscrape: adds support for PuppetDB service discovery
This commit adds support for
[PuppetDB](https://www.puppet.com/docs/puppetdb/8/overview.html) service
discovery to the `vmagent` and `victoria-metrics-single` components.

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5744
2024-10-27 20:38:34 +01:00
Andrii Chubatiuk
7e60afb6fc
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:36:33 +01:00
Dan Dascalescu
66971d3141
Fix "loosing" typo in README.md (#7368)
### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/)
2024-10-27 00:56:42 -07:00
Zhu Jiekun
cd2222aa95
dashboards: fix query for full ETA vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes (#7355)
Some checks failed
publish-docs / Build (push) Has been cancelled
### Describe Your Changes

Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7334

available disk space should be 
```
(vm_free_disk_space_bytes{job=~...} - vm_free_disk_space_limit_bytes{job=~...})
```
instead of 
```
vm_free_disk_space_bytes{job=~...}
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-25 15:09:14 +02:00
Hui Wang
0172e65b8d
docs: clarify flags -search.maxxxDuration can only be overridden to… (#7227)
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
CodeQL JS/TS / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
… a smaller value with `timeout` arg
2024-10-25 11:11:09 +02:00
Github Actions
cf344f5250
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@ed19341 (#7352) 2024-10-25 01:18:06 -07:00
Github Actions
2224424136
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@9e6db28 (#7350) 2024-10-25 01:16:42 -07:00
Yury Molodov
dd89745a34
vmui/logs: fix query and limit update issue (#7294)
### Describe Your Changes

Fixes issues with incorrect updating of query and limit fields, and
resolves the problem where the display tab resets.

Related issue: #7279 and #7290

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-25 09:32:20 +02:00