Commit Graph

4929 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
f9e23bf8e3
lib/logstorage: add join pipe for joining multiple query results 2024-11-06 18:53:29 +01:00
Aliaksandr Valialkin
3a5f1019ba
app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing 2024-11-06 18:53:28 +01:00
Evgeniy Negriy
d27dfac5c6
app/vmselect: fixes graphite function transformRemoveEmptySeries
Previously it incorrectly applied xFilesFactor, if it's value equal to 0.

 This commit properly handles this case and returns result according to
the graphite documentation:
  
`xFilesFactor follows the same semantics as in Whisper storage schemas. Setting it to 0 (the default) means that only a single value in the series needs to be non-null for it to be considered non-empty, setting it to 1 means that all values ​​in the series must be non-null. A setting of 0.5 means that at least half the values ​​in the series must be non-null.`

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Evgeniy Negriy <einegriy@avito.ru>
2024-11-06 17:35:59 +01:00
Zhu Jiekun
3aeb1b96a2
app/vlinisert/loki: properly parse json logs with structured metadata
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously,  loki request json parsers rejected log line if tuple size != 2.

This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the  log message stream.


related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-06 17:25:05 +01:00
Andrii Chubatiuk
a88f896b43
promql: exclude limit_offset from default by metric name sorting (#7402)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
### Describe Your Changes

I don't like this solution, but it works. Other possible solutions
described in an issue

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7068

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-11-06 15:10:23 +01:00
hagen1778
05f2e9548d
docs/victorialogs: recommend using separate installations of vmalert for vm and vl
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-06 14:58:11 +01:00
hagen1778
a5f1764171
docs/victorialogs: clarify usage of -rule.defaultRuleType=vlogs
User experience suggests that examples shouldn't have `-rule.defaultRuleType=vlogs` set,
as it may confuse users who run vmalert with their existing rules or only use
rules from examples for testing purposes.

This change is supposed to remove the confusion by removing `-rule.defaultRuleType=vlogs`
from default recommendations and explcitily specifying `type` on group level in examples.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-06 14:55:58 +01:00
Roman Khavronenko
0390d58a34
docs: fix typos in vlogs rules examples (#7457)
* fix typos in rules definition. Otherwise, they can't pass validation
* add code types for rendered examples

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-06 13:55:08 +01:00
Zhu Jiekun
f16a58f14c
vmctl: fixed import duplicate data when query result contains multiple series (#7330)
### Describe Your Changes

Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301

When querying with condition like `WHERE a=1` (looking for series A),
InfluxDB can return data with the tag `a=1` (series A) and data with the
tag `a=1,b=1` (series B).

However, series B is will be queried later and it's data should not be
combined into series A's data.

This PR filter those series that are not identical to the original query
condition.

For table `example`:
```
// time                           host    region  value
// ----                           ----    ------  -----
// 2024-10-25T02:12:13.469720983Z serverA us_west 0.64
// 2024-10-25T02:12:21.832755213Z serverA us_west 0.75
// 2024-10-25T02:12:32.351876479Z serverA         0.88
// 2024-10-25T02:12:37.766320484Z serverA         0.95
```

The query for series A (`example_value{host="serverA"}`) and result will
be:
```SQL
SELECT * FROM example WHERE host = "serverA"
```
```json
{
	"results": [{
		"statement_id": 0,
		"series": [{
			"name": "cpu",
			"columns": ["time", "host", "region", "value"],
			"values": [
				["2024-10-25T02:12:13.469720983Z", "serverA", "us_west", 0.64],
				["2024-10-25T02:12:21.832755213Z", "serverA", "us_west", 0.75],
				["2024-10-25T02:12:32.351876479Z", "serverA", null, 0.88],
				["2024-10-25T02:12:37.766320484Z", "serverA", null, 0.95]
			]
		}]
	}]
}
```

We need to abandon `values[0]` and `values[1]` because the value of
**unwanted** column `region` is not null.

As for series B (`example_value{host="serverA", region="us_west"}`), no
change needed since the query filter out unwanted rows already.

### Note
This is a draft PR for verifying the fix.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-11-06 13:53:49 +01:00
Artem Navoiev
115ac7d0d7
docs: understand your setup size fix formula
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-11-06 12:29:50 +01:00
Artem Fetishev
d212243a0f
docs: Fix images used in Query data docs (#7443)
### Describe Your Changes

This is a follow up for #7435. Images need to be updated too:

-   The time is changed from 10 hrs to 08 hrs
-   A missing data point is added to the range query image
-   Source escalidraw has been updated as well

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-11-06 11:58:56 +01:00
Github Actions
021f514842
Automatic update helm docs from VictoriaMetrics/helm-charts@f598096 (#7452)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-06 11:57:34 +01:00
Zhu Jiekun
11d735a91f
docs: [puppetdb] add changelog for puppetdb service discovery (#7455)
### Describe Your Changes

Add puppetdb sd to changelog of `v1.106.0` version.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-06 09:19:51 +01:00
Github Actions
fa7adcaeaf
Automatic update operator docs from VictoriaMetrics/operator@f8ca70f (#7444)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-11-05 19:05:33 +01:00
Github Actions
74a9c6f91c
Automatic update helm docs from VictoriaMetrics/helm-charts@76f3195 (#7447)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: zekker6 <1367798+zekker6@users.noreply.github.com>
2024-11-05 19:05:24 +01:00
Zakhar Bessarab
be677065bc
{docker,docs}: update references to latest release
Update references to latest release - v1.106.0.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-05 18:40:13 +01:00
Smaine Kahlouch
5b838b03ec
docs(articles): add blog ogenki to third party articles (#7414)
### Describe Your Changes

Adding a blog post that introduces VictoriaMetrics to third party
articles

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Smaine Kahlouch <smainklh@gmail.com>
2024-11-05 18:30:48 +01:00
Andrii Chubatiuk
e0930687f1
vlinsert: support datadog logs
This commit adds the following changes:

- Added support to push datadog logs with examples of how to ingest data
using Vector and Fluentbit
- Updated VictoriaLogs examples directory structure to have single
container image for victorialogs, agent (fluentbit, vector, etc) but
multiple configurations for different protocols

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632
2024-11-05 16:52:35 +01:00
Arie Heinrich
2e8f420d84
docs: spelling fixes (#7420)
### Describe Your Changes

Christmas is early and you get the first present in the shape of
spelling fixes.
Sorry for the big amount :)

### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-05 16:44:23 +01:00
Github Actions
397997b2a8
Automatic update helm docs from VictoriaMetrics/helm-charts@f8ad8eb (#7438)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-05 16:36:34 +01:00
Artem Fetishev
d311c12dce
documentation: Fix query docs
Some checks are pending
publish-docs / Build (push) Waiting to run
This commit changes the following:
-   The datetime has been fixed so it corresponds to the timestamps in example samples. The datetime now also include the UTC time zone and is changed to adhere ISO format.
-   The data points in query range result have been fixed to match the inserted data.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-11-05 15:36:14 +01:00
Zakhar Bessarab
a3401c8d64
docs/changelog: sync latest release
- remove reference to sparse cache as it was reverted in 9f9cc24e4c
- add reference to 1.102.6 and 1.97.11 LTS releases

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-05 10:08:36 -03:00
Roman Khavronenko
f52577a1a8
docs: mention VM version when DS filters became available
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-05 13:31:27 +01:00
Zakhar Bessarab
371e193279
docs/CHANGELOG.md: cut v1.106.0
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
CodeQL JS/TS / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 11:13:39 -03:00
Zakhar Bessarab
9f9cc24e4c
Revert "lib/mergeset: add sparse indexdb cache (#7269)"
This reverts commit 837d0d136d.
2024-11-04 10:29:14 -03:00
Aliaksandr Valialkin
5733e56e40
docs/VictoriaLogs/querying: refer to vlogscli from docs for /select/logsql/query and /select/logsql/tail HTTP endpoints
Some checks failed
publish-docs / Build (push) Has been cancelled
These endpoints are easier to query via vlogscli
2024-11-02 12:31:40 +01:00
Aliaksandr Valialkin
37a58677fa
docs/VictoriaLogs/data-ingestion: mention Grafana Agent and Grafana Alloy additionally to Promtail as log collectors for Grafana Loki 2024-11-02 11:36:22 +01:00
Aliaksandr Valialkin
c7e242b8ce
docs/VictoriaLogs/LogsQL.md: remove misleading information that min and max stats functions work only for numeric values
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
These functions work for both numeric and string values after ad505a7a9a
2024-11-01 20:25:33 +01:00
Aliaksandr Valialkin
a86df52adf
docs/VictoriaLogs/LogsQL.md: typo fix in the link to description for sum_len stats function: sum-len-stats -> sum_len-stats
This is a follow-up for 364f084b43
2024-11-01 20:13:47 +01:00
Aliaksandr Valialkin
4478e48eb6
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445
2024-11-01 20:06:17 +01:00
Aliaksandr Valialkin
b3e2db5647
docs/VictoriaLogs/keyConcepts.md: remove duplicate sentence about missing timezone information in the ingested _time field values 2024-11-01 17:20:28 +01:00
Andrii Chubatiuk
5e2cb78cce
Updated k8s guides (#7411)
### Describe Your Changes

- updated dependencies versions 
- removed Helm v2 from docs
- fixed VMAgent configuration

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-01 19:13:59 +04:00
f41gh7
5cc2e49297
app/vmgateway: fixes rate limit for multitenant requests
Some checks failed
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
CodeQL JS/TS / Analyze (push) Has been cancelled
Previously vmgateway returned error for the requests with multitenant
tenant.

 This commit allows to rate limit multitenant requests and apply global
rate limit for it.

Currently it supports only queries for rate limiting.

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7201

This commit also addresses gateway start-up crash if datasource.url is not accessible.

Previously vmgateway could crash at start-up with enabled rate limiting if datasource for metrics
was not avaiable for any reason.  It seems, that crash is expected. But in fact it's not. For instance, datasource could be in restart phase.

Replaces crash with log message error. It increased availability of vmgateway component.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-10-31 20:20:02 +01:00
Nikolay
bba08f7846
lib/promscrape: add relabel configs to global section
This commit adds `metric_relabel_configs` and `relabel_configs` fields
into the `global` section of scrape configuration file.

 New fields are used as global relabeling rules for the scrape targets.

 These relabel configs are prepended to the target relabel configs.
This feature is useful to:
* apply global rules to __meta labels from service discovery targets.
* drop noisy labels during scrapping.
* mutate labels without affecting metrics ingested via any of push
protocols. 

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6966

---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 19:58:22 +01:00
hagen1778
06621995bd
docs: mention stats object in Prometheus API enhancements
The doc explains fields meaning in `stats` object.
It also clarifies their purpose.
See related ticket https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7170

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 15:09:04 +01:00
Github Actions
955f3660de
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3ecaebd (#7395) 2024-10-31 14:08:35 +01:00
hagen1778
c5b36138e2
docs: mention #7392 in changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 14:08:08 +01:00
Viet Hung Nguyen
21d1385ae1
vmalert-tool: set default interval for unittest input_series (#7392)
### Describe Your Changes
Currently it is not optional option, but if user forgot to set, it
defaults to 0, which cause unexpected behavior.

This change sets default = evaluation_interval similar to promtool
behavior.


https://prometheus.io/docs/prometheus/2.55/configuration/unit_testing_rules/#test_group
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Viet Hung Nguyen <hvn@familug.org>
2024-10-31 14:04:50 +01:00
Roman Khavronenko
3f0e2ab3b2
deployment/alerts: add RemoteWriteDroppingData to vmalert rules (#7393)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 14:03:08 +01:00
Yury Molodov
bfb55d5f2f
vmui: fix the display of the link to vmalert (#7380)
### Describe Your Changes

Fix the issue mentioned in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7088#issuecomment-2391360368

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-31 13:43:58 +01:00
Aliaksandr Valialkin
24b6e117dd
deployment/docker: update VictoriaLogs from v0.39.0-victorialogs to v0.40.0-victorialogs
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.40.0-victorialogs
2024-10-30 23:43:20 +01:00
Aliaksandr Valialkin
1a7b55009b
docs/VictoriaLogs/CHANGELOG.md: cut v0.40.0-victorialogs release 2024-10-30 23:38:06 +01:00
Aliaksandr Valialkin
f88e2ae9fb
docs/VictoriaLogs/README.md: follow-up for 0f6b9e9490: consistently use canonical url for Grafana dashboards
See also 0a5ffb3bc1
2024-10-30 23:29:23 +01:00
Aliaksandr Valialkin
0f6b9e9490
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886
2024-10-30 23:20:36 +01:00
Aliaksandr Valialkin
c5d08d317c
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Aliaksandr Valialkin
2e635a42d8
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Aliaksandr Valialkin
f62502a943
docs/VictoriaLogs/querying/README.md: typo fix after f2cd284cf4a7429848fb4be1522483a9b93a43e7: Alls -> All
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
2024-10-30 19:08:57 +01:00
Aliaksandr Valialkin
7603446850
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365
2024-10-30 17:59:49 +01:00
Artem Fetishev
683f8c2780
dashboards: add Restarts panel (#7394)
Reopening PR #7373 from a branch in VictoriaMetrics repo in order to
enable edits and rebase.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 16:44:08 +01:00
hagen1778
6494606924
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 16:06:57 +01:00
Aliaksandr Valialkin
25bca94d04
deployment/docker: update VictoriaLogs from v0.38.0-victorialogs to v0.39.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.39.0-victorialogs
2024-10-30 15:11:17 +01:00
Aliaksandr Valialkin
13822f6d11
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release 2024-10-30 15:06:00 +01:00
Aliaksandr Valialkin
16ee470da6
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.
2024-10-30 14:59:38 +01:00
Aliaksandr Valialkin
96466562b6
docs/VictoriaLogs/data-ingestion/README.md: formatting fixes after 8b36529b32
- Remove leading whitespace from the first lines in 'HTTP parameters' chapter.
  This whitespace isn't needed for the markdown formatting.

- Add leading whitespace for the second sentence in the list bullet describing AccountID and ProjectID HTTP headers.
  This fixes markdown formatting for this list bullet.
2024-10-30 14:23:54 +01:00
Aliaksandr Valialkin
ed73f8350b
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.
2024-10-30 14:17:33 +01:00
Aliaksandr Valialkin
102e9d4f4e
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.
2024-10-30 13:39:28 +01:00
Dan Dascalescu
258ee93fd1
docs: clarify "single" in Single-server-VictoriaMetrics.md (#7369)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
### Describe Your Changes

"Single version" is unclear, since VM is also a single-executable. I
think "single-node" is clearer.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-30 12:55:52 +01:00
danish-mehmood
8dc4e2b5a5
docs: fix typos and format in case study (#7374)
### Describe Your Changes

- made small typo fix in case studies

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-30 12:48:09 +01:00
hagen1778
cfba770c8e
docs: consistently update twitter.com to x.com
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7383

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:44:50 +01:00
Github Actions
b2cf8685e5
Automatic update helm docs from VictoriaMetrics/helm-charts@87ea94d (#7385)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-10-30 12:39:35 +01:00
hagen1778
2c93353173
docs: add frequently asked questions to vmalert integration with vlogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:38:47 +01:00
hagen1778
b789a9dc83
docs: mark alerting feature as done in VictoriaLogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:20:00 +01:00
hagen1778
77b690ab26
docs: update version placeholder to point to the next version
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 12:19:44 +01:00
Artem Navoiev
3473440d26
docs: logs vmalert add identifier
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-10-30 09:45:28 +01:00
hagen1778
2ac07aa813
docs: rm unused vmanomaly assets
Some checks failed
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
CodeQL JS/TS / Analyze (push) Has been cancelled
These images copies are present in vmanomaly subfolder.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:52:25 +01:00
hagen1778
a49eb7d4f5
docs: add step for updating VM version in docs
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:50:40 +01:00
hagen1778
b86b0dd910
docs: update VM versions to the latest version
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 18:48:36 +01:00
Aliaksandr Valialkin
c963d7d10d
docs/VictoriaLogs/CHANGELOG.md: remove unnededed with prefix in front of rank at top pipe example 2024-10-29 18:27:40 +01:00
Aliaksandr Valialkin
c43a6ce0eb
deployment/docker: update VictoriaLogs from v0.37.0-victorialogs to v0.38.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.38.0-victorialogs
2024-10-29 18:16:16 +01:00
Aliaksandr Valialkin
12223cf5d0
docs/VictoriaLogs/CHANGELOG.md: cut v0.38.0 release 2024-10-29 18:08:36 +01:00
Aliaksandr Valialkin
7a623c225f
lib/logstorage: follow-up for af831a6c906158f371f1b6810706fa0a54b78386
Sync the code between top and sort pipes regarding the code related to rank.
2024-10-29 16:44:46 +01:00
Aliaksandr Valialkin
3c06d083ea
lib/logstorage: add an ability to return rank from top pipe results 2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
7a62eefa34
lib/logstorage: dynamically adjust the number of (bloom, values) shards in a part depending on the number of non-const columns
This allows reducing the amounts of data, which must be read during queries over logs with big number of fields (aka "wide events").
This, in turn, improves query performance when the data, which needs to be scanned during the query, doesn't fit OS page cache.
2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
67b4059aa4
docs/VictoriaLogs/README.md: add tuning chapter 2024-10-29 16:44:44 +01:00
Aliaksandr Valialkin
8d968acd0a
lib/logstorage: avoid reading columnsHeader data when field_values pipe is applied directly to log filters
This improves performance of `field_values` pipe when it is applied to large number of data blocks.
This also improves performance of /select/logsql/field_values HTTP API.
2024-10-29 16:44:44 +01:00
Hui Wang
68bad22fd2
vmalert: integrate with victorialogs (#7255)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.

Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.

Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.

Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 16:30:39 +01:00
Fred Navruzov
5d73b8b866
docs/vmanomaly - release 1.18.0 (#7378)
Some checks are pending
publish-docs / Build (push) Waiting to run
### Describe Your Changes

docs/vmanomaly - release 1.18.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-28 17:25:24 +02:00
Zakhar Bessarab
4e50d6eed3
lib/storage/partition: prevent panic in case resulting in-memory part is empty after merge (#7329)
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
It is possible for in-memory part to be empty if ingested samples are
removed by retention filters. In this case, data will not be discarded
due to retention before creating in memory part. After in-memory parts
merge samples will be removed resulting in creating completely empty
part at destination.

 This commit checks for resulting part and skips it, if it's empty.

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-27 20:40:13 +01:00
Zhu Jiekun
f06c7e99fe
lib/promscrape: adds support for PuppetDB service discovery
This commit adds support for
[PuppetDB](https://www.puppet.com/docs/puppetdb/8/overview.html) service
discovery to the `vmagent` and `victoria-metrics-single` components.

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5744
2024-10-27 20:38:34 +01:00
Andrii Chubatiuk
7e60afb6fc
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:36:33 +01:00
Dan Dascalescu
66971d3141
Fix "loosing" typo in README.md (#7368)
### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/)
2024-10-27 00:56:42 -07:00
Zhu Jiekun
cd2222aa95
dashboards: fix query for full ETA vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes (#7355)
Some checks failed
publish-docs / Build (push) Has been cancelled
### Describe Your Changes

Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7334

available disk space should be 
```
(vm_free_disk_space_bytes{job=~...} - vm_free_disk_space_limit_bytes{job=~...})
```
instead of 
```
vm_free_disk_space_bytes{job=~...}
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-25 15:09:14 +02:00
Hui Wang
0172e65b8d
docs: clarify flags -search.maxxxDuration can only be overridden to… (#7227)
Some checks failed
publish-docs / Build (push) Waiting to run
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
CodeQL JS/TS / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
… a smaller value with `timeout` arg
2024-10-25 11:11:09 +02:00
Github Actions
cf344f5250
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@ed19341 (#7352) 2024-10-25 01:18:06 -07:00
Github Actions
2224424136
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@9e6db28 (#7350) 2024-10-25 01:16:42 -07:00
Yury Molodov
dd89745a34
vmui/logs: fix query and limit update issue (#7294)
### Describe Your Changes

Fixes issues with incorrect updating of query and limit fields, and
resolves the problem where the display tab resets.

Related issue: #7279 and #7290

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-25 09:32:20 +02:00
Zakhar Bessarab
372ce74d62
docs/guides-vmgateway-grafana: update guide (#7347)
Some checks are pending
publish-docs / Build (push) Waiting to run
### Describe Your Changes

- update to recent versions of components
- add information about the license key
- add example configuration for remote write with oAuth identity for
vmagent

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-24 23:01:57 +04:00
Andrii Chubatiuk
fc537bea00
lib/promscrape/discovery/kubernetes: support kubernetes native sidecars (#7324)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
This commit adds Kubernetes Native Sidecar support. 

It's the special type of init containers, that have restartPolicy == "Always" and continue to run after container initialization. 


related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7287
2024-10-24 17:04:12 +02:00
Zakhar Bessarab
837d0d136d
lib/mergeset: add sparse indexdb cache (#7269)
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182

- add a separate index cache for searches which might read through large
amounts of random entries. Primary use-case for this is retention and
downsampling filters, when applying filters background merge needs to
fetch large amount of random entries which pollutes an index cache.
Using different caches allows to reduce effect on memory usage and cache
efficiency of the main cache while still having high cache hit rate. A
separate cache size is 5% of allowed memory.

- reduce size of indexdb/dataBlocks cache in order to free memory for
new sparse cache. Reduced size by 5% and moved this to a separate cache.

- add a separate metricName search which does not cache metric names -
this is needed in order to allow disabling metric name caching when
applying downsampling/retention filters. Applying filters during
background merge accesses random entries, this fills up cache and does
not provide an actual improvement due to random access nature.


Merge performance and memory usage stats before and after the change:

- before

![image](https://github.com/user-attachments/assets/485fffbb-c225-47ae-b5c5-bc8a7c57b36e)


- after

![image](https://github.com/user-attachments/assets/f4ba3440-7c1c-4ec1-bc54-4d2ab431eef5)

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-24 15:21:17 +02:00
Andrii Chubatiuk
5fecb77f69
app/vmctl: fix match expression for vm-native protocol with --vm-native-disable-per-metric-migration flag enabled (#7310)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7309

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-24 14:57:58 +02:00
hagen1778
53b7288e0d
docs: clarify that auto generated metrics can't be relabeled in scrape config
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-24 12:09:30 +02:00
Github Actions
ec0abe736a
Automatic update helm docs from VictoriaMetrics/helm-charts@f3aac52 (#7335)
Some checks are pending
publish-docs / Build (push) Waiting to run
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
2024-10-24 12:20:06 +08:00
Github Actions
6434fa2c4e
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@12fc7d5 (#7340)
Some checks are pending
publish-docs / Build (push) Waiting to run
2024-10-23 13:02:18 -07:00
hagen1778
7e53324f5d
docs: clarify that vminsert also supports exponential histogram parsing
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-23 11:47:50 +02:00
Github Actions
5ee3bc98d6
Automatic update helm docs from VictoriaMetrics/helm-charts@6237358 (#7327)
Some checks are pending
publish-docs / Build (push) Waiting to run
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-10-23 03:27:32 +04:00
Roman Khavronenko
0204ce942d
app/vmalert: update -remoteWrite.concurrency and -remoteWrite.flushInterval (#7272)
Some checks are pending
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
Auto-adjust `-remoteWrite.concurrency` cmd-line flags with the number of
available CPU cores in the same way as vmagent does. With this change
the default behavior of vmalert in high-loaded installation should
become more resilient. This change also reduces
`-remoteWrite.flushInterval` from `5s` to `2s` to provide better data
freshness.


---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-10-22 14:43:55 +02:00
Antoine Deschênes
d656934d22
vmalert: properly set group_name and file fields for recording rules (#7298)
This commit properly adds `group_name` and `file` fields for recording rules web api response   at `/api/v1/rules`.
Previously these fields were blank.

Related issue https://github.com/victoriaMetrics/victoriaMetrics/issues/7297

Signed-off-by: Antoine Deschênes <antoine.deschenes@linux.com>
2024-10-22 14:13:56 +02:00
Github Actions
ac82b5aea6
Automatic update helm docs from VictoriaMetrics/helm-charts@999d44f (#7316)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-10-22 14:04:48 +02:00
Github Actions
e8c7d6373e
Automatic update operator docs from VictoriaMetrics/operator@b357f60 (#7319)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-10-22 14:04:24 +02:00
Fred Navruzov
b17fce3e4b
docs/vmanomaly-release-1.17.2 (#7322)
### Describe Your Changes

- release 1.17.2 updates
- added sections on logging and CLI args to docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-22 13:42:37 +03:00