Commit Graph

4716 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
f5dfe1cacd
lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe
Previously only logs inside the selected time range could be returned by stream_context pipe.
For example, the following query could return up to 10 surrounding logs only for the last 5 minutes,
while most users expect this query should return up to 10 surrounding logs without restrictions on the time range.

    _time:5m panic | stream_context before 10

This enables the ability to implement stream context feature at VictoriaLogs web UI: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7063 .

Reduce memory usage when returning stream context over big log streams with millions of entries.
The new logic scans over all the log messages for the selected log stream, while keeping in memory only
the given number of surrounding logs. Previously all the logs for the given log stream on the selected time range
were loaded in memory before selecting the needed surrounding logs.
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6730 .

Reduce the scan performance for big log streams by fetching only the requested fields. For example, the following
query should be executed much faster than before if logs contain many fields other than _stream, _msg and _time:

    panic | stream_context after 30 | fields _stream, _msg, _time
2024-09-26 17:04:39 +02:00
Aliaksandr Valialkin
4d27933041
app/vlinsert: support _time field without timezone information during data ingestion
Use local timezone of the host server in this case. The timezone can be overridden
with TZ environment variable if needed.

While at it, allow using whitespace instead of T as a delimiter between data and time
in the ingested _time field. For example, '2024-09-20 10:20:30' is now accepted
during data ingestion. This is valid ISO8601 format, which is used by some log shippers,
so it should be supported. This format is also known as SQL datetime format.

Also assume local time zone when time without timezone information is passed to querying APIs.
Previously such a time was parsed in UTC timezone. Add `Z` to the end of the time string
if the old behaviour is preferred.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6721
2024-09-26 12:50:14 +02:00
Aliaksandr Valialkin
e63d50e0c0
docs/VictoriaLogs/CHANGELOG.md: typo fix: itentifying -> identifying 2024-09-26 09:41:50 +02:00
Zhu Jiekun
3fa72b2c1b
feature: [victorialogs] drop logs without non-empty _msg field (#7056)
### Describe Your Changes

VictoriaLogs allows logs without `_msg` field or `_msg` field is empty.
This lead to incorrect search result. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785

This pull request search for non-empty `_msg` field before log entry is
added to `LogRows`.

New counter `vl_rows_dropped_total{reason="msg_not_exist"}` is
introduced.

Example log output:
```
2024-09-23T02:33:19.719Z        warn    app/vlinsert/insertutils/common_params.go:189   dropping log line without _msg field; [{@timestamp 2024-09-18T13:42:16.600000000Z} {Attributes.array.attribute ["many","values"]} {Attributes.boolean.attribute true} {Attributes.double.attribute 637.704} {Attributes.int.attribute 10} {Attributes.map.attribute.some.map.key some value} {Attributes.string.attribute some string} {Body Example ddddddddddlog record} {Resource.service.name my.service} {Scope.my.scope.attribute some scope attribute} {Scope.name my.library} {Scope.version 1.0.0} {SeverityNumber 10} {SeverityText Information} {SpanId eee19b7ec3c1b174} {TraceFlags 0} {TraceId 5b8efff798038103d269b633813fc60c}]
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] Benchmark for potential performance loss.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-26 09:35:58 +02:00
Aliaksandr Valialkin
92885f99dd
docs/VictoriaLogs/CHANGELOG.md: document the fix for Windows build
This is a follow-up for 264c2ec6bd

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6973
2024-09-26 09:17:23 +02:00
Aliaksandr Valialkin
15db8d3c47
docs/VictoriaLogs/CHANGELOG.md: typo fix after 255d1d4e13: returns -> return 2024-09-26 09:01:01 +02:00
Aliaksandr Valialkin
3a556bd15a
app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop
Previously the original timestamp was used in the copied query, so _time:duration filters
were applied to the original time range: (timestamp-duration ... timestamp]. This resulted
in stopped live tailing, since new logs have timestamps bigger than the original time range.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7028
2024-09-26 08:57:48 +02:00
Github Actions
b5ca6717ca
Automatic update operator docs from VictoriaMetrics/operator@5271a59 (#7099)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-26 08:57:48 +02:00
Aliaksandr Valialkin
cac886dfcf
docs/LTS-releases.md: consistently use v prefix in front of VictoriaMetrics releases 2024-09-25 19:29:43 +02:00
Aliaksandr Valialkin
55ecf4f766
lib/logstorage: add blocks_count pipe
This pipe is useful for debugging purposes when the number of processed blocks must be calculated for the given query:

    <query> | blocks_count

This helps detecting the root cause of query performance slowdown in cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070
2024-09-25 19:18:38 +02:00
Aliaksandr Valialkin
246c339e3d
lib/logstorage: read timestamps column when it is really needed during query execution
Previously timestamps column was read unconditionally on every query.
This could significantly slow down queries, which do not need reading this column
like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070 .
2024-09-25 19:18:37 +02:00
Andrii Chubatiuk
1824dd378f
docs/victorialogs/data-ingestion: removed FluentBit Elasticsearch from examples (#7093)
removed FluentBit Elasticsearch example from docs as custom headers are
not supported by elasticsearch output till
https://github.com/fluent/fluent-bit/pull/9416 is merged and released

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6985

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit f934f71708)
2024-09-25 18:47:24 +02:00
Andrii Chubatiuk
73afeaf510
deployment: restructure victorialogs examples (#6971)
### Describe Your Changes

- Use common compose.yaml file for all victorialogs setups to set
version in a single place and override it on demand for each agent and
protocol
- Replaced multiple victorialogs instances in HA setup with single setup
with `deploy.replica` parameter set
- Added fluentd setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit e75ae1b274)
2024-09-25 18:39:43 +02:00
Github Actions
5c95c0be26
Automatic update operator docs from VictoriaMetrics/operator@1feab7d (#7092)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit 612be0954c)
2024-09-25 17:07:27 +02:00
Roman Khavronenko
e716e5904f
app/vmalert: bump default values for sending data to remoteWrite.url (#7084)
* `remoteWrite.maxQueueSize` from `100_000` to `1_000_000`, this should
improve resiliency of recording rules that produce many series;
* `remoteWrite.maxBatchSize` from `1_000` to `10_000`, this should be
more efficient to send from netwroking perspective;
* `remoteWrite.concurrency` from `1` to `4`, this should imrpove speed
of sending the generated series.

The new settings should improve remote write performance of vmalert with
default settings.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
(cherry picked from commit 6b1b47df54)
2024-09-25 17:07:27 +02:00
Roman Khavronenko
9c87c5c7cc
docs: update CONTRIBUTING.md with practical requirements (#7087)
The change supposed to have more practical recommendations and reflect
the real processes for maintaining the project.


Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-24 18:38:54 +02:00
Github Actions
76c74ec57f
Automatic update operator docs from VictoriaMetrics/operator@27ad7e1 (#7086)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit e28265fa39)
2024-09-24 16:58:17 +02:00
Zhu Jiekun
73ae5dcfc5
vmagent: remote write respect Retry-After in header (#6124)
### Describe Your Changes
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6097

#### Changed
- Remote write retry policy in `vmagent` is changed into:
  1. Respect `Retry-After` duration if exists.
2. Otherwise, calculate next retry duration by backoff policy (x2) and
max retry duration limit.

#### Docs
- `CHANGELOG.md`.

---
### Checklist
The following checks are mandatory:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Zakhar Bessarab <me@zekker-dev.tk>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5319acb8ed)
2024-09-24 16:58:16 +02:00
Dmytro Kozlov
869b09122a
lib/promscrape: show only unhealthy targets if show_only_unhealthy filter is enabled (#6960)
### Describe Your Changes

It is better to show only unhealthy targets instead of all of them when
`show_only_unhealthy` filter is enabled.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3536

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit cbeb7d50e8)
2024-09-24 16:58:16 +02:00
Phuong Le
7f05b9f88f
docs: update logos files and usage rules (#6980)
### Describe Your Changes

New logos and usage guideline

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit df665a13c9)
2024-09-24 16:58:16 +02:00
Zhu Jiekun
143d137265
docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074)
### Describe Your Changes

By default, the `elasticsearchexporter` in OTel Collector puts the log
message under a field other than `_msg` (e.g., `Body`). Without
specifying via an HTTP header, those logs may not be queried correctly.
See also:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785.

This pull request updates the example configuration and notes for the
`elasticsearchexporter`.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit fea4433362)
2024-09-24 16:58:16 +02:00
Github Actions
b810af5cae
Automatic update operator docs from VictoriaMetrics/operator@75bc1b4 (#7080)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 524579d9bd)
2024-09-24 16:58:15 +02:00
Roman Khavronenko
deb2f87074
deployment: add panel and alerts for displying go scheduler latency (#7078)
The panel and alerting rule should help to understand whether VM
component doesn't have enough CPU resources or gets throttled. The alert
is applicable for all VM components.
The panel was added to vmalert, vmagent, vmsingle, vm clusert and
victorialogs dashes.

-------------------

This alerting rule should have help us identify resource shortage for
sandbox vmagent - see [this
link](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.range_input=23d13h25m25s424ms&g0.end_input=2024-09-23T14%3A11%3A00&g0.relative_time=none&g0.tab=0&g0.expr=histogram_quantile%280.99%2C+sum%28rate%28go_sched_latencies_seconds_bucket%7Bjob%3D%22vmagent-monitoring-vmagent%22%7D%5B5m%5D%29%29+by+%28le%2C+job%2C+instance%29%29+%3E+0.1)
for example. We weren't aware of resource shortage, because VM metrics
assumed this vmagent had 1vCPU while in fact its limit was 0.2vCPU.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4d0b41e63b)
2024-09-24 16:58:14 +02:00
Aliaksandr Valialkin
2a17cddf3d
app/vmselect/promql: consistently replace NaN data points with non-NaN values for range_first and range_last functions
It is expected that range_first and range_last functions return non-nan const value across all the points
if the original series contains at least a single non-NaN value. Previously this rule was violated for NaN data points
in the original series. This could confuse users.

While at it, add tests for series with NaN values across all the range_* and running_* functions, in order to maintain
consistent handling of NaN values across these functions.
2024-09-23 15:00:05 +02:00
hagen1778
9ebc056861
docs: add note about testing new releases on testing env
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-23 15:00:05 +02:00
Zakhar Bessarab
85b74a453e
license: add ability to reload keys (#775)
* lib/license: add support of license key hot-reload

* docs: add info about license key hot reload

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-09-23 09:04:33 +02:00
Aliaksandr Valialkin
3dd4af8f78
docs/changelog/CHANGELOG.md: moved the description of the fix for proper usage of -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval from FEATURE to BUGFIX section
The previous behaviour was incorrect, since it is unexpected that the -streamAggr.dedupInterval
and -remoteWrite.streamAggr.dedupInterval is applied to processed samples only if -streamAggr.config isn't set.

This is a follow-up for d523015f27
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6711
2024-09-23 08:56:55 +02:00
Github Actions
73206473ff
Automatic update helm docs from VictoriaMetrics/helm-charts@428cb36 (#7072)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-23 08:56:54 +02:00
Aliaksandr Valialkin
34165eae0f
docs/changelog/CHANGELOG.md: document bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7009
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7064
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7009

This is a follow-up for 55febc0920
2024-09-22 21:58:13 +02:00
Github Actions
c299b22c39
Automatic update operator docs from VictoriaMetrics/operator@d661554 (#7069)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-22 21:58:13 +02:00
Artem Navoiev
b35f3f3e2c
docs: cloud siplify menu items names for notification, user managemenr and notifs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:38:56 +02:00
Github Actions
8f5e0ebbb4
Automatic update helm docs from VictoriaMetrics/helm-charts@f0e007f (#7059)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-20 17:38:55 +02:00
Artem Navoiev
1aea528746
docs add root menu to _index.md as homepage
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:38:04 +02:00
Github Actions
32f4557179
Automatic update operator docs from VictoriaMetrics/operator@fdd3f9f (#7065)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: tenmozes <1381404+tenmozes@users.noreply.github.com>
2024-09-20 17:38:04 +02:00
Andrii Chubatiuk
3b4afff8ba
docs: updated root menu items (#7061)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:38:03 +02:00
Artem Navoiev
9dac0782b9
docs: set operator menu item weight to 30
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:35:28 +02:00
Artem Navoiev
c7662aff6b
docs: clarify operator name.2
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:35:27 +02:00
Artem Navoiev
19911bb458
docs: clarify operator name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 17:35:27 +02:00
Hui Wang
d2b98245ea
vmalert: fix variable $activeAt value when templating rule annotation in replay mode
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-09-20 17:34:54 +02:00
hagen1778
e3a1eaab87
docs: fix more typos in the changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 17:34:28 +02:00
hagen1778
a99cf73eac
docs: rm update node about loggerMaxArgLen as it doesn't have incompatibility effect
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 17:34:27 +02:00
hagen1778
5b1015de4c
docs: fix typo in link in change line about NaN
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 17:33:43 +02:00
Thomas Danielsson
3e5b80244a
docs: fix typo in the changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 17:33:43 +02:00
Yury Molodov
5a905e2e94
vmui: change the query_range request method from GET to POST (#7039)
### Describe Your Changes

change the `/query_range` and `/query` requests method from `GET` to
`POST`. See #6288.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b0bdb92729)
2024-09-19 15:48:09 +02:00
Roman Khavronenko
e6dac18db3
lib/logger: increase default value of -loggerMaxArgLen cmd-line fla… (#7008)
…g from 1e3 to 5e3

This should improve visibility on errors produced by very long queries.

The change is classified as BUG in order to port it to LTS releases.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Mathias Palmersheim <mathias@victoriametrics.com>
(cherry picked from commit e115b85770)
2024-09-19 15:48:09 +02:00
Yury Molodov
868fa988b3
vmui/logs: add auto refresh (#7038)
### Describe Your Changes

Add auto refresh
#7017

![image](https://github.com/user-attachments/assets/20ed1102-d5e4-4d3f-9c24-7d298d93400a)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit bc9cb69170)
2024-09-19 15:48:08 +02:00
Aliaksandr Valialkin
7e0ff1ee46
app/vlselect/logsql: call Query.Optimize() on the cloned query in order to replace * filter with filterNoop inside getLastNQueryResults()
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785

(cherry picked from commit e86891b010)
2024-09-19 15:48:07 +02:00
Aliaksandr Valialkin
4e00e4428e
app/vmselect/promql: properly calculate c1 and c2 and c1 or c2 by upgrading github.com/VictoriaMetrics/metricsql to v0.79.0
The fix is in the https://github.com/VictoriaMetrics/metricsql/pull/34
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6637

(cherry picked from commit b82e2cabc5)
2024-09-19 15:48:06 +02:00
Github Actions
8e5e772af4
Automatic update operator docs from VictoriaMetrics/operator@fe90f57 (#7026)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit 8bc30b68cd)
2024-09-19 15:48:06 +02:00
Dmytro Kozlov
ab86aad292
docs/victoriametrics-cloud: updated pictures and descriptions with new functionality (#7014)
Improved VictoriaMetrics documentation for cloud

Related issue: https://github.com/VictoriaMetrics/cloud/issues/2143

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 4c228f1e18)
2024-09-19 15:48:06 +02:00
Alexander Marshalov
c0e764d4a4
[vmcloud]: Cloud tiers assumptions docs (#6948)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit c0272463d9)
2024-09-19 15:46:53 +02:00
Artem Navoiev
bcab5861ae
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 277fed9990)
2024-09-19 15:46:52 +02:00
Artem Navoiev
1ad6e8df74
docs: adjust image size
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 153cceb124)
2024-09-19 15:46:52 +02:00
Artem Navoiev
382444aebb
docs: fix links to docker
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit ae5f98f46b)
2024-09-19 15:46:52 +02:00
Artem Navoiev
947b9d3e8d
docs: victorialogs just mention telegraf as we support not only http but more
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 71df11dfbb)
2024-09-19 15:46:51 +02:00
Artem Navoiev
c073206f0c
docs: update roadmap, mark telegraf and otel as done and link the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 47cd7bb4c1)
2024-09-19 15:46:51 +02:00
f41gh7
65798f0f8d
docs/changelog: mention vmagent kafka consumer bugfix
Changes were made to the enteprise repository
2024-09-19 15:36:04 +02:00
Nikolay
6f99dcc7c1
lib/storage: consistently check for missing metricID index records (#6967)
* Previously, only metricID->metricName missing index records were
tracked with deadline But it was possible a case for missing
metricID->TSID index records. IndexDB metrics fix exposed misleading
metric for such missing records.

* This commit adds check for metricID->TSID missing index records. And
delete missing metricID entry if it hit 60 second deadline.

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6931

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-16 13:07:37 +02:00
Nikolay
c32032ac1b
lib/fs: properly call windows APIs (#6998)
Previously we manually imported system windows DDLs
and made direct syscall.

 But golang exposes syscall wrappers with sys/windows package.
It seems, that direct syscall was broken at 1.23 golang release. It was
`GetDiskFreeSpace` syscall in our case.

This commit replaces all manual syscalls with wrappers

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6973

Related golang issue:
https://github.com/golang/go/issues/69029

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-13 13:19:04 +02:00
Dima Lazerka
465c7ad045
docs: fixes misspelled typos
Also tried to make it catch "Authorisation" in the future, fixed a lot
of other misspells along the way, but didn't make it catch
"Authorisation" anyway.

- Fix misspelled "Authorization" header name
- Fix misspelled "organization"
- Fix more misspells
2024-09-13 13:19:03 +02:00
Aliaksandr Valialkin
213cd045fc
docs/VictoriaLogs/querying/README.md: typo fixes: use field_values instead field_names where needed 2024-09-11 09:42:42 +02:00
Hui Wang
c7fc0d0d2f
vmalert: do not send message to alertmanager when alert has no label … (#6823)
…pair

`alert_relabel_configs` in [notifier
config](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file)
can drop alert labels when used to filter different tenant alert message
to different notifier.
alertmanager would report error like `msg="Failed to validate alerts"
err="at least one label pair required"` in this case, but the rest of
the alerts inside one request would still be valid in alertmanager, so
it's not severe.

(cherry picked from commit ae4d376e41)
2024-09-09 16:06:44 +02:00
Artem Navoiev
d5cfce693a
docs: Add VictoriaLogs HA example for single nodes (#6945)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit f9a8c09fe8)
2024-09-09 16:06:43 +02:00
hagen1778
30f34b28d6
docs: mention -dryRun flag next to config example in vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7983ea4edb)
2024-09-09 16:06:43 +02:00
Aliaksandr Valialkin
c68c7c0d1a
deployment: update VictoriaLogs from v0.28.0-victorialogs to v0.29.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.29.0-victorialogs
2024-09-08 21:23:29 +02:00
Aliaksandr Valialkin
32c0e02958
docs/VictoriaLogs/CHANGELOG.md: cut v0.29.0 2024-09-08 21:11:34 +02:00
Aliaksandr Valialkin
8eef397d29
deployment/docker: update base Alpine docker image from 3.20.2 to 3.20.3
See https://alpinelinux.org/posts/Alpine-3.17.10-3.18.9-3.19.4-3.20.3-released.html
2024-09-08 19:27:05 +02:00
Aliaksandr Valialkin
4b49b62a58
lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions 2024-09-08 12:28:33 +02:00
Aliaksandr Valialkin
c448189f69
app/vlselect: add /select/logsql/stats_query_range endpoint for building time series panels in VictoriaLogs plugin for Grafana
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6943
Updates https://github.com/VictoriaMetrics/victorialogs-datasource/issues/61
2024-09-07 00:44:34 +02:00
Aliaksandr Valialkin
01c8e12370
app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706
2024-09-06 23:00:58 +02:00
Aliaksandr Valialkin
e90e809c00
deployment: update Go builder from Go1.23.0 to Go1.23.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.1+label%3ACherryPickApproved
2024-09-06 22:57:56 +02:00
Aliaksandr Valialkin
c5badeea08
lib/logstorage: substitute | operator with or operator at math pipe
This is needed for avoiding confusion between the `|` operator at `math` pipe and `|` pipe delimiter.
For example, the following query was parsed unexpectedly:

   * | math foo / bar | fields x

as

   * | math foo / (bar | fields) as x

Substituting `|` with `or` inside `math` pipe fixes this ambiguity.
2024-09-06 22:43:29 +02:00
f41gh7
b9d9aad85a
docs/changelog: mention storage changes
After a5424e95b3

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-06 18:13:48 +02:00
Zakhar Bessarab
e4b8b82901
Vmgateway no prefix string (#784)
* app/vmgateway: allow skipping Bearer prefix, parsing access as string

- allow disabling of "Bearer" prefix check - This is needed in order to support OIDC systems where identity token is provided separately from access token and it does not contain "Bearer" prefix(such as Azure Entra ID, ex AD).a

- support parsing "vm_access" claim as a string - This is helpful for systems where claims can only be mapped to string.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: mention vmgateway updates

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-09-06 16:20:29 +02:00
Aliaksandr Valialkin
7dcce1ca02
lib/logstorage: pre-calculate hashes from tokens used in bloom filter search
Previously per-token hashes for per-block bloom filters were re-calculated on every scanned block.
This could be slow when the number of tokens is big or when the number of blocks to scan is big.
Pre-calculate hashes for bloom filters and then use them for searching in bloom filters.
This improves performance by 2.5x for in(...) filters with many values to search inside `in()`.
2024-09-05 19:44:42 +02:00
f41gh7
6fe0a2700e
docs/changelog: mention storage NaN changes
follow-up after 39294b4919

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-05 16:58:10 +02:00
Zhu Jiekun
8848614315
lib/discovery/azure: fix host check in next link in Azure SD (#6915)
Previous bugfix at 49f63b2 only partially fixed pagination host validation error.

 Before this fix it was:
```
unexpected nextLink host \"management.azure.com\", expecting \"https://management.azure.com\"
```

Now we only check the `Host` without schema. 

However, when Azure respond `nextLink` in `Host:Port` format, the
`nextLink` check will fail:
```
unexpected nextLink host \"management.azure.com:443\", expecting \"management.azure.com\"
```

This pull request further relaxes the checks by only checking the
`Hostname`.

---

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6912
2024-09-05 16:58:10 +02:00
Hui Wang
9cb1704d3c
lib/storage: fix metric vm_object_references{type="indexdb"} (#6937)
follow up
4ecc370acb

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-05 16:57:48 +02:00
Aliaksandr Valialkin
5763a957ef
lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields
Previously (f1:foo OR f2:bar) was incorrectly returning `foo` token for `f1` and `bar` token for `f2`.
These tokens were used for checking against bloom filter for every data block, so the data block,
which didn't contain simultaneously `foo` token for `f1` field and `bar` token for `f2` field, was skipped.
This was incorrect, since such a block may contain logs matching the original OR filter.

The fix is to return common tokens from `OR`-delimted filters only if these tokens exist at EVERY such filter
for the given field name. If some `OR`-delimited filter misses the given field name, then `OR`-delimited filters
do not contain common tokens, which could be used for checking against bloom filter.

While at it, add more tests covering various edge cases for filters delimited by AND and OR.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6554
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6556
2024-09-05 16:57:47 +02:00
Artem Navoiev
3fa0c17df1
docs: remove old png file as we have webp now
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-05 16:57:47 +02:00
Github Actions
2150382566
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@edd3259 (#6946) 2024-09-05 16:57:47 +02:00
hagen1778
06de23d23b
docs: fix typo in vmagent.md
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-05 16:57:47 +02:00
f41gh7
3e0bfb2b38
docs/changelog: mention enterprise changes
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-04 15:39:55 +02:00
f41gh7
ddae38c583
docs/changelog: moves victorialogs changes to proper file
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-04 15:39:55 +02:00
Roman Khavronenko
ef83ec9c21
docs: clarify why 1.102.1 was publicly available (#6933)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-04 15:39:55 +02:00
hagen1778
e9964eef64
docs: bump last LTS versions
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-04 15:39:55 +02:00
Andrii Chubatiuk
711f2cc4f2
vlinsert: added opentelemetry logs support
Commit adds the following changes:

* Adds support of OpenTelemetry logs for Victoria Logs with protobuf encoded messages

*  json encoding is not supported for the following reasons:
   - It brings a lot of fragile code, which works inefficiently.
   - json encoding is impossible to use with language SDK.

* splits metrics and logs structures at lib/protoparser/opentelemetry/pb package.

* adds docs with examples for opentelemetry logs.

---
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4839

Co-authored-by: AndrewChubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 20:24:01 +02:00
f41gh7
dcc525b388
follow-up after 1731c0eabf
* updates change log
* adds VL-Debug http header
* updates doc
* extracts only the first value of http headers for VL-Stream-Fields and VL-Ignore-Fields.
  It makes behaviour the same as Query string args. And allows to easily configure client applications.
  Since most of the client collectors don't support multi value headers.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 20:24:01 +02:00
Andrii Chubatiuk
d5fe4566e5
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 20:24:00 +02:00
hagen1778
665e59e23a
dashboards/vmagent: fix legend captions for stream aggregation related panels.
Before they were displaying wrong label names.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-03 20:24:00 +02:00
Hui Wang
a21aea5dd4
stream aggregation: perform deduplication for all received data when … (#6711)
…specifying `-streamAggr.dedupInterval` or
`-remoteWrite.streamAggr.dedupInterval` command-line flag

[The
documentation](https://docs.victoriametrics.com/stream-aggregation/)
contains conflicting descriptions regarding deduplication for
non-matched series when `-remoteWrite.streamAggr.config` and / or
`-streamAggr.config` are set:
1. Statement below says **all the received data** is deduplicated:
>[vmagent](https://docs.victoriametrics.com/vmagent/) supports
relabeling, deduplication and stream aggregation for all the received
data, scraped or pushed. Then, the collected data will be forwarded to
specified -remoteWrite.url destinations. The data processing order is
the following:
>1. all the received data is relabeled according to the specified
[-remoteWrite.relabelConfig](https://docs.victoriametrics.com/vmagent/#relabeling)
(if it is set)
>2. all the received data is deduplicated according to specified
[-streamAggr.dedupInterval](https://docs.victoriametrics.com/stream-aggregation/#deduplication)
(if it is set to duration bigger than 0)

2. Another statement says the deduplication is performed individually
for the **matching samples**
>The de-deduplication is performed after applying
[relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) and
before performing the aggregation. If the -remoteWrite.streamAggr.config
and / or -streamAggr.config is set, then the de-duplication is performed
individually per each [stream aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config)
for the matching samples after applying
[input_relabel_configs](https://docs.victoriametrics.com/stream-aggregation/#relabeling).

Considering the following deduplication use cases:
1. To apply deduplication(globally or for specific remoteWrite
destination) for all the received data, scraped or pushed
--- using `-streamAggr.dedupInterval` or
`-remoteWrite.streamAggr.dedupInterval`.
2. To deduplicate and aggregate metrics that match the rule `match`
filters
--- using `-remoteWrite.streamAggr.config` and specifiying
`dedup_interval` option in [stream aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
3. To deduplicate all the received data while having `streamAggr.config`
for some metrics
--- no way for a single vmagent now, need to set up two level vmagents

This PR implements case3.

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit d523015f27)
2024-09-03 10:49:38 +02:00
Github Actions
d37505661f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@1f6f9ea (#6925)
(cherry picked from commit 78067ff2c2)
2024-09-03 10:49:31 +02:00
Github Actions
4db80f51f3
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@297faf8 (#6926)
(cherry picked from commit fd5469e039)
2024-09-03 10:49:31 +02:00
jackyin
66789a8144
lib/logstorage: and filter results in unexpected response (#6556)
fix #6554
andfilter shouldn't return orfilter field which result in bloomfilter
return false.

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 975ed27a76)
2024-09-03 10:49:25 +02:00
Github Actions
7ea1e932ff
Automatic update operator docs from VictoriaMetrics/operator@7eaac41 (#6923)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: tenmozes <1381404+tenmozes@users.noreply.github.com>
(cherry picked from commit f102b14ac9)
2024-09-03 10:49:23 +02:00
hagen1778
de2713ec01
docs: fix broken links to prev changelogs
The prev links like `/changelog_2021/`
stopped working after 9dc8d1debd
because these files now require specifying the parent `changelog` in the path, like `/changelog/changelog_2021/`.

This fix adds an alias for an old link.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9d94573572)
2024-09-03 10:49:20 +02:00
Ivan Yatskevich
e7eab095d0
docs/vmalert: mention VM Cloud where it can be helpful to a user (#6888)
### Describe Your Changes

Add mentions of VictoriaMetrics Cloud to the documentation of vmalert
where this info is helpful to a user.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 8a8a1d5df2)
2024-08-30 11:17:27 +02:00
Ivan Yatskevich
44b0fd8805
docs/quickstart: describe steps to run VM on VM Cloud (#6877)
### Describe Your Changes

Describe steps to run VictoriaMetrics Single node or Cluster on
VictoriaMetrics Cloud
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f6278d99dd)
2024-08-30 11:17:27 +02:00
Zhu Jiekun
046f52df6e
docs: fix incorrect URLs for resetCache and OTel guide (#6906)
### Describe Your Changes

This pull request fixes incorrect URLs in two places:

1. In the OTel guide, which has been corrected in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6880, but one
incorrect URL is still missing.
2. In the URL example, the cache reset endpoint for vmselect / Cluster
version is `/internal/resetRollupResultCache`, but it is mistakenly
noted as `/select/internal/resetRollupResultCache`, which misguides the
user. (introduced in #4468)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit f572365a93)
2024-08-30 11:17:27 +02:00
hagen1778
8568b88870
docs/guides: fix images in vmgateway guide
Follow-up after ce4cc4cbb2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b27c4b198f)
2024-08-30 11:17:26 +02:00
hagen1778
d605c8de37
docs/release-guide: rm RPM packages from the list
RPM packages are now managed by community at https://github.com/VictoriaMetrics-Community/victoriametrics-rpm

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 11ae873d0a)
2024-08-30 11:17:26 +02:00
Github Actions
9bddc8998e
Automatic update operator docs from VictoriaMetrics/operator@f82db30 (#6910)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit f42ef2152f)
2024-08-30 11:17:26 +02:00
hagen1778
ac6f82e119
docs/victorialogs/CHANGELOG.md: mention issue #4750 in the change
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 00d09d67dc)
2024-08-30 11:17:25 +02:00
hagen1778
bd6e5a23bb
docs/CHANGELOG.md: update changelog with LTS release notes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d5755e55ef)
2024-08-30 11:17:25 +02:00
Github Actions
52540bd6e0
Automatic update operator docs from VictoriaMetrics/operator@bf5e160 (#6901)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit 982c907ee1)
2024-08-30 11:17:19 +02:00
hagen1778
b7329adb38
docs/CHANGELOG.md: cut v1.103.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5aeb759df9)
2024-08-28 13:48:55 +02:00
hagen1778
b036d78008
docs: pre-release doc update
* typo fix
* mention version starting from features are available

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e71cfdcfa5)
2024-08-28 13:48:55 +02:00
hagen1778
5699ee901a
docs: update dedup docs for cluster
* remove repeating sentences
* clarify why identical dedup config is important

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d9982520a8)
2024-08-28 13:38:27 +02:00
f41gh7
7686f42abe
docs/changelog: mention bugfix
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-28 11:51:18 +02:00
Nikolay
0f9536eaf5
lib/storage: properly add previous indexDB metrics (#6890)
Previously, some extIndexDB metrics were not registered. It resulted
into missing metrics, if metric value was added to the extIndexDB. It's
a usual case for search requests at both indexes.

 Current commit updates all metrics from extIndexDB according to the
current IndexDB. It must fix such cases

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6868

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 4ecc370acb)
2024-08-28 11:17:23 +02:00
Cuong Le
bba68c3a30
VictoriaLogs/keyConcepts.md: fix broken anchor to #other-fields (#6896)
### Describe Your Changes

The anchor to "Other fields" section should be #other-fields (instead of
#other-field)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Cuong Le <cuongleqq@gmail.com>
(cherry picked from commit 6154bc9466)
2024-08-28 11:17:22 +02:00
Zhu Jiekun
51988ad45d
docs: Add backup and restore doc for VictoriaLogs
This commit adds doc for VictoriaLogs backup and restore based on `rsync`.
2024-08-27 23:08:29 +02:00
rtm0
4c31a6a1fc
lib/storage: properly handle maxMetrics limit at metricID search
`TL;DR` This PR improves the metric IDs search in IndexDB:

- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search

A unit test has been added to ensure that this refactoring does not
break anything.

---

Function calls before the fix:

```
idb.searchMetricIDs
    |__ is.searchMetricIDs
        |__ is.searchMetricIDsInternal
            |__ is.updateMetricIDsForTagFilters
                |__ is.tryUpdatingMetricIDsForDateRange
                |                       |
                |__ is.getMetricIDsForDateAndFilters
```

- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.

Problems with this solution:

1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.

Solution:

1.  Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in  `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-08-27 23:08:17 +02:00
rtm0
b51c6bf75d
lib/storage: properly register index records with RegisterMetricNames
Once the timeseries is in tsidCache, new entries won't be created in
per-day index because the RegisterMetricNames() code does consider
different dates for the same timeseries. So this case has been added.

The same bug exists for AddRows() but it is not manifested because the
index entries are finally created in updatePerDateData().

RegisterMetricNames also updated to increase the newTimeseriesCreated
counter because it actually creates new time series in index.

A unit tests has been added that check all possible data patterns
(different metric names and dates) and code branches in both
RegisterMetricNames and AddRows. The total number of new unit tests is
around 100 which increaded the running time of storage tests by 50%.

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2024-08-27 23:00:27 +02:00
hagen1778
6db4682d95
docs: typo fix
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6f17ee0d0f)
2024-08-27 15:49:46 +02:00
rtm0
291ff0497c
docs: document IndexDB (#6840)
### Describe Your Changes

This is an attempt to document IndexDB. I guess I was trying to touch
the important points that might be of interest for the end users while
refraining from making it too detailed (such as I did not enumerate and
describe all the specific record types).

Please take a look and any suggestions are very welcome.

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
(cherry picked from commit e7f1297517)
2024-08-27 15:44:08 +02:00
Zhu Jiekun
5b49fd83be
lib/promrelabel: follow-up for 8958cecad6
In the previous commit 8958cecad6
the default ports (80/443) were removed for both the `scrapeURL` and
`instance` label values for those targets without a port in
`__address__`. Different values in the `instance` label generate new
time series.

This commit reverts the changes made to the `instance` label. Now,
for those targets:
- `scrapeURL` will remain unchanged.
- The `instance` label value will include the default port.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6792
(cherry picked from commit e97e966f82)
2024-08-27 15:44:07 +02:00
Zhu Jiekun
397318f8f3
docs: fix example links in getting started with OpenTelemetry (#6880)
### Describe Your Changes

Fix broken link on:
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/index.html#building-the-go-application-instrumented-with-metrics
- https://docs.victoriametrics.com/guides/app.go-collector.example
- https://docs.victoriametrics.com/guides/app.go.example

The valid links **could be**:
-
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/app.go-collector.example
-
https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/app.go.example

Also went throught `/guides` folder and did not found doc with same
issue. Please advice if you know more.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 9c3b5116dd)
2024-08-27 15:44:07 +02:00
Fred Navruzov
259f49089a
docs: vmanomaly - release notes v1.15.[6-9] (#6879)
### Describe Your Changes

release notes for patches 1.15.6 - 9

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit ac485a8486)
2024-08-27 15:44:07 +02:00
Github Actions
76d4171d71
Automatic update operator docs from VictoriaMetrics/operator@3ba727f (#6875)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit e7b3a996da)
2024-08-26 14:48:30 +02:00
YuDong Tang
cab3ef8294
app/vmselect:add command-line flag -search.inmemoryBufSizeBytes (#6869)
add command-line flag `-search.inmemoryBufSizeBytes` for configuring size of in-memory buffers used by vmselect during processing of vmstorage responses. A new summary metric `vm_tmp_blocks_inmemory_file_size_bytes` is exposed to show the size of the buffer during requests processing. 

The new setting can be used by experienced users to adjust memory usage by vmselect when processing
many small read requests. Instead of allocating 4MB buffers each time, vmselect can be instructed to lower
the buffer size via `-search.inmemoryBufSizeBytes`. To make the decision whether this flag needs to be adjusted
users can consult with `vm_tmp_blocks_inmemory_file_size_bytes` which shows the actual size of buffers used
during query processing.

----------

The detailed information of this PR can be found in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6851

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-26 14:37:45 +02:00
Github Actions
02d314107e
Automatic update operator docs from VictoriaMetrics/operator@369ab8d (#6866)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit 84ca4d18f0)
2024-08-23 15:28:44 +02:00
Yury Akudovich
f759371c00
app/vmagent: add remoteWrite.retryMinInterval and remoteWrite.retryMaxTime flags (#6289)
## Describe Your Changes

Add RemoteWrite Retry Controls

This PR introduces two new flags to the remote write functionality:
- remoteWrite.retryMinInterval
- remoteWrite.retryMaxTime

These flags provide finer control over the retry behavior for
remoteWrite operations, allowing users to customize the minimum interval
between retries and the maximum duration for retry attempts.

Fixes #5486.

## Checklist

- [x] The following checks are mandatory:

My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Yury Akudovich <ya@matterlabs.dev>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d0f5a9d77a)
2024-08-23 15:28:44 +02:00
Denys Holius
e24ebe3e54
docs/guides/guide-delete-or-replace-metrics/README.md: adds a link to the API examples (#6863)
### Describe Your Changes

Adds a link to the API example section that describes how to delete
metrics on VM Single.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 336406e2e1)
2024-08-23 15:28:44 +02:00
Roman Khavronenko
c6e0780f4b
app/vmalert: update parsing for instant responses (#6859)
This change is made in attempt to reduce memory usage by vmalert when
parsing big instant responses from VM/Prometheus.

In
a5c427bac4
vmalert switched from std json lib to fastjson lib in order to reduce
amount of allocations, as according to highloaded profiles of vmalert
the CPU is mostly spent on GC.

But switching to fastjson resulted into excessive memory usage for cases
when vmalert has to parse long json lines, which usually happens when
instant response contains many `metric` objects.

In this change we do a mixed parsing:
1. Slice of `metric` objects is parsed with std lib to keep mem low
2. Each `metric` object is parsed with fastjson to reduce allocs

The benchmark results are the following:
```
pkg: github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource
BenchmarkParsePrometheusResponse/Instant_std+fastjson-10                    1760            668959 ns/op          280147 B/op       5781 allocs/op
MBs allocated at heap: 493.078392
mallocs: 18655472
BenchmarkParsePrometheusResponse/Instant_fastjson-10                        6109            198258 ns/op          172839 B/op       5548 allocs/op
MBs allocated at heap: 1056.384464
mallocs: 34457184
BenchmarkParsePrometheusResponse/Instant_std-10                             1287            950987 ns/op          451677 B/op       9619 allocs/op
MBs allocated at heap: 580.802976
mallocs: 13351636
```
The benchmark function code with mem measurement is available here
https://gist.github.com/hagen1778/b9c3ca7f8ca7d6b21aec9777112c5810

The benchmark contains 3 results:
1. Instant_std+fastjson is the implementation in this change
2. Instant_fastjson-10 is the implementation from
a5c427bac4
3. BenchmarkParsePrometheusResponse/Instant_std-10 is implementation
before
a5c427bac4

According to these results, this new implementation is slower than
previous, but faster than before switching to fastjson. It also has
lower number of allocations and roughly the same memory allocation on
heap with GC turned off.

---------

Other changes:
1. rm BenchmarkMetrics as it doesn't measure anything
2. simplify BenchmarkParsePrometheusResponse into
BenchmarkPromInstantUnmarshal

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 23:56:11 +02:00
Yury Molodov
a3509add4d
vmui: add column search in table settings (#6804)
### Describe Your Changes

Add search functionality to the column display settings in the table
 #6668

![image](https://github.com/user-attachments/assets/e9bd52c3-6428-4d4f-8b7f-d83dd80b6912)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e35237920a)
2024-08-22 16:59:18 +02:00
hagen1778
d8266777d6
docs: fix ordering in vmagent docs
Looks like auto-numeration doesn't work if item in the list
takes more than one line.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit fa9de9b45c)
2024-08-22 16:59:18 +02:00
hagen1778
4251d8073f
docs: fix root page in keyConcepts
Add _index.md to fix `http://docs/keyconcepts` link.
Without this change only `http://docs/keyconcepts/keyconcepts` was working.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit dbb4899d71)
2024-08-22 16:59:17 +02:00
Roman Khavronenko
f0e5df7b89
docs: move keyconcepts to separate dir (#6855)
Moving key-concepts-related docs to a separate dir should make it easier
to navigate in `docs/` folder and helps to avoid adding prefixes to
image assets.

-----------

The change shouldn't have any visual changes or changes to the links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 22975d28ae)
2024-08-22 16:59:17 +02:00
Roman Khavronenko
b3c99579ea
docs: move changelog to dir (#6853)
Moving changelog-related docs to a separate dir should make it easier to
navigate in `docs/` folder.

-----------

The change shouldn't have any visual changes or changes to the links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9dc8d1debd)
2024-08-22 16:59:17 +02:00
hagen1778
d1bc2f53f3
docs: rm mentions of google group as it was deprecated
See https://docs.victoriametrics.com/#community-and-contributions

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 74e8d6ad55)
2024-08-21 14:14:40 +02:00
Github Actions
a3f2321da4
Automatic update operator docs from VictoriaMetrics/operator@5fd7d6e (#6845)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit a82dda1d18)
2024-08-21 14:14:39 +02:00
Github Actions
89a4fb3355
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@c23c4d1 (#6844)
(cherry picked from commit 37654e0604)
2024-08-21 14:14:39 +02:00
Zakhar Bessarab
4cda699972
make: reset permissions when creating release archive (#6846)
### Describe Your Changes

Forcefully set permissions for release binaries to 1000:1000. This helps
to avoid issues such as
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6788 on
systems with limitations around UID:GID configuration.

"1000" UID and GID is widely used by linux distributions for the first
user in the system.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 654c1cb9d1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 14:14:33 +02:00
Ayush Chauhan
4abc5b9045
docs: add zomato case study (#6848)
(cherry picked from commit af54ddea23)
2024-08-21 14:12:07 +02:00
Dima Lazerka
f1325531a1
vmui: Fix initial serverUrl for vmanomaly (#6834)
- fix TS lint
- anomaly: remove /vmui
- anomaly: minor inspections fix
- docs: fix broken links to headings

### Describe Your Changes

Initially vmanomaly opened with `/vmui` in serverUrl, remove it.

(cherry picked from commit 535a9ed059)
2024-08-21 14:12:07 +02:00
hagen1778
c348ab860e
docs: move change from VM changelog to VL changelog
The change for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6764
was added to VM changelog by mistake

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2dba4165e8)
2024-08-20 17:16:14 +02:00
jackyin
98758ef18e
vmui: fix not found index.js in VictoriaLogs (#6770)
fix #6764
the index.js file is for [this
feature](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#predefined-dashboards),
the feature is just for victoriametrics. so the index.js is deleted in
victorialogs.
i just add an empty index.js to fix it.

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3ebdd3bcb8)
2024-08-20 17:14:00 +02:00
Nikolay
08cbbf8134
lib/promscrape: fixes proxy autorization (#6783)
* Adds custom dial func for HTTP-Connect and socks5 proxy tunnels.
  Standard golang http.transport exposes GetProxyConnectHeader function,
  but it doesn't allow to use separate tls config for proxy.
  It also not possible to enforce HTTP-Connect with standard http lib.
* For http scrape targets, by default http.Transport.Proxy function must
  be used. Since it has special case with full uri forward.
* Adds proxy.URL json methods that allow to properly copy internal
fields, like User/Password.
It should fix bug with proxy_url. When credentials specified at URL was
ignored.
* Adds tests for scrape client proxy requests

related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6771
2024-08-19 22:50:39 +02:00
Zhu Jiekun
8958cecad6
lib/promrelabel: stop adding default port 80/433 to address label
*  It was necessary to add default ports for fasthttp client. After migration to the std.httpclient it's no longer needed.
* An additional configuration is required at proxy servers with implicitly set 80/443 ports to the host header (such as HA proxy.

It's expected that after upgrade __address_ label may change. But it should be rare case. 80/443 ports are not widely used at monitoring ecosystem. And it shouldn't have much impact. 

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6792

Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-08-19 22:50:39 +02:00
Hui Wang
e8f5dbd598
vmalert: add command line flag -notifier.headers (#6751)
to allow configuring additional headers in each request to the
corresponding notifier.
Other flags like `-datasource.headers`, `-remoteWrite.headers` already
use `^^` as delimiter, it's consistent to use it in `-notifier.headers`
as well.

related https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260
vmalert can integrate with alertmanager that supports multi-tenant by
adding tenantID header`X-Scope-OrgID` in requests.
In multitenancy, vmalert can also filter alerts which send to different
notifier addresses(or with different header settings) using
`alert_relabel_configs`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0f1ec33892)
2024-08-19 21:41:57 +02:00
Github Actions
efb6a070c0
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@de3f649 (#6837)
(cherry picked from commit 0ef59bf7b3)
2024-08-19 21:41:57 +02:00
Hui Wang
9edd42127d
vmalert-tool: add -external.label and -external.url command-line … (#6766)
…flags to perform the same as vmalert

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6735

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0fc1130f47)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-19 21:41:56 +02:00
Andrii Chubatiuk
71632bb4b6
docs: reuse hugo image for webp conversion (#6825)
Use same hugo docker image for webp conversion.
While there, remove unused *.png images.

(cherry picked from commit 36bc458e9e)
2024-08-19 17:45:41 +02:00
Fred Navruzov
6c4948bd35
docs: vmanomaly - v1.15.5 patch notes (#6835)
### Describe Your Changes

docs: vmanomaly - v1.15.5 patch notes

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 0e464a3a4f)
2024-08-19 17:45:41 +02:00
Daria Karavaieva
fa820bd9cf
docs/vmanomaly: updated model list in Overview (#6832)
### Describe Your Changes

Updated model list in Anomaly Detection Overview

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 7279899a8a)
2024-08-19 17:45:41 +02:00
Github Actions
c57e68a0cd
Automatic update operator docs from VictoriaMetrics/operator@64879fb (#6831)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 015f0b0424)
2024-08-19 17:45:40 +02:00
Roman Khavronenko
d4240c4a3e
lib/httputils: parse URL before creating HTTP transport (#6820)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6740

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-16 11:34:49 +02:00
Fred Navruzov
d373608c41
docs: vmanomaly - release notes for 1.15.4 (#6813)
### Describe Your Changes

release notes for 1.15.4 patch

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-16 11:34:49 +02:00
Zakhar Bessarab
84b8ea7337
app/vmseleсt/promql: fix calculation of histogram buckets
This issue was introduced in 6a4bd5049b

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6714

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-08-15 10:13:54 +02:00
Daria Karavaieva
e2f384edfe
docs/vmanomaly:change links from relative to absolute (#6809)
### Describe Your Changes

- change links from relative to absolute under Anomaly Detection section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-15 10:13:54 +02:00
Fred Navruzov
124fbd5081
docs/vmanomaly - changelog update to v1.15.3 (#6808)
### Describe Your Changes

changelog updates to v1.15.3 patch of `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-15 10:13:54 +02:00
Fred Navruzov
dfe6d0920d
docs: vmanomaly - reader page update (#6806)
### Describe Your Changes

small update to `data_range` parameter in uppermost config conversion
example

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-15 10:13:53 +02:00
Fred Navruzov
bd80dd2ce1
docs: vmanomaly - release v1.15.2 (#6802)
### Describe Your Changes

update vmanomaly docs to forthcoming release v1.15.2

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-15 10:13:53 +02:00
Nikolay
f255800da3
app/vminsert: returns back memory optimisation (#6794)
Production workload shows that it's useful optimisation.

Channel based objects pool allows to handle irregural data ingestion
requests and make memory allocations more smooth.
It's improves sync.Pool efficiency, since objects from sync.Pool removed
after 2 GC cycles. With GOGC=30 value, GC runs significantly more often.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6733

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 10:49:09 -04:00
ccliu
8729052623
vmagent: resolve the issue where usePromCompatibleNaming is not working (#6776)
Describe Your Changes
When I use usePromCompatibleNaming with vmagent to process data that
needs to be formatted from different sources such as InfluxDB, I find
that it doesn’t work

However, it works in vminsert. I found that vminsert uses the
HasRelabeling method to determine whether to relabel.
```go
func HasRelabeling() bool {
	pcs := pcsGlobal.Load()
	return pcs.Len() > 0 || *usePromCompatibleNaming
}
```
in vmagent, the decision to relabel is determined only by
pcsGlobal.Len() > 0. However, in the applyRelabeling method, the
usePromCompatibleNaming logic is also used to determine whether to
relabel in the error handling.
```go
func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, pcs *promrelabel.ParsedConfigs) []prompbmarshal.TimeSeries {
	if pcs.Len() == 0 && !*usePromCompatibleNaming {
		// Nothing to change.
		return tss
	}
```
So I think that the logic for determining whether to relabel in vmagent
is not as expected.

Checklist
The following checks are mandatory:

[]My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit d134a310f3)
2024-08-13 10:33:55 -04:00
jackyin
11233364b6
vlogs: add select/deselect all button to table settings in UI (#6680)
fix #6668, just add **select all** and "unselect all" func.

https://github.com/user-attachments/assets/0c31385b-def0-4618-aa9c-5ba4bb6f56c3

---------

Co-authored-by: Yury Molodov <yurymolodov@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f5bc46b3e)
2024-08-13 10:33:54 -04:00
Zhu Jiekun
27a6be6630
docs: add more details to -cacheDataPath vmselect flag (#6708)
vmselect will create `./tmp` dir under `cacheDataPath`. If
`cacheDataPath` is set to `/`, vmselect will use `/tmp`.

content under `/tmp` dir might be auto removed based on the OS
behaviour. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5770

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 09:17:43 -04:00
Hui Wang
e74d5f266e
stream aggregation: do not allow to enable -stream.keepInput and `k… (#6723)
…eep_metric_names` options in stream aggregation config together

With aggregated data and raw data under the same metric, results would
be confusing.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 62d19369a3)
2024-08-13 09:08:27 -04:00
hagen1778
440b34fa77
docs: mention https://blog.zomato.com/migrating-to-victoriametrics-a-complete-overhaul-for-enhanced-observability
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 331573b0bb)
2024-08-13 09:08:27 -04:00
Fred Navruzov
b084b4fb0f
docs/vmanomaly - typos fix & clarity (#6793)
### Describe Your Changes

typos fix & clarity improvement of vmanomaly docs after v1.15.1 release

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit a99fcfbf1a)
2024-08-13 09:08:27 -04:00
Fred Navruzov
b673fe28e9
docs: vmanomaly - release v1.15.1 (#6782)
### Describe Your Changes

 vmanomaly - release v1.15.1 updates to docs:
- changelog page
- reader page (new arguments docs)
- typos & fixes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 985e4f0b99)
2024-08-13 09:08:27 -04:00
Zhu Jiekun
49f63b2b9a
app/vmagent: fixes azure service discovery pagination
Azure API response with link to the next page was incorrectly validate. Validation used url.Host header to match configure API URL.


https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6784
2024-08-09 15:26:18 +02:00
hagen1778
229f8217a0
docs: mention deduplication change in update notes for 1.100.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-09 15:26:18 +02:00
Dmytro Kozlov
76c5fa00bd
docs: update user management guide for cloud (#6738)
### Describe Your Changes

Updated user management guide with new cloud content
This PR should be merged after the cloud PR

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-09 15:26:18 +02:00
Zakhar Bessarab
54315fbad6
lib/backup/s3remote: add retryer configuration (#6747)
### Describe Your Changes

This helps to improve reliability of performing backups in environments
with unreliable connection and tolerate temporary errors at S3 provider
side.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6732

Default retry timeout is up to 3 minutes to make this consistent with
the same configuration for GCS:
a05317f61f/lib/backup/gcsremote/gcs.go (L70-L76)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit cb00b4b00f)
2024-08-07 16:59:23 +02:00
Roman Khavronenko
c41a9b8d17
lib/bytesutil: smooth buffer growth rate (#6761)
Before, buffer growth was always x2 of its size, which could lead to
excessive memory usage when processing big amount of data.
For example, scraping a target with hundreds of MBs in response could
result into hih memory spikes in vmagent because buffer has to double
its size to fit the response. See
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759

The change smoothes out the growth rate, trading higher allocation rate
for lower mem usage at certain conditions.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f28f496a9d)
2024-08-07 16:59:23 +02:00
Andrii Chubatiuk
77c3bbf3fc
docs: updated guides structure, removed deprecated sort option (#6767)
### Describe Your Changes

* `sort` param is unused by the current website engine, and was present only for compatibility
with previous website engine. It is time to remove it as it makes no effect
* re-structure guides content into folders to simplify assets management

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 35d77a3bed)
2024-08-07 16:59:22 +02:00
Fred Navruzov
c75dcc91ad
docs/vmanomaly: fix typos after v1.15.0 (#6774)
### Describe Your Changes

Fixing remaining typos and missing words after v.1.15.0 updates

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 264e9caf5a)
2024-08-07 16:59:22 +02:00
Fred Navruzov
0becae4ad4
docs/vmanomaly: release v1.15.0 (#6768)
### Describe Your Changes

- updated docs on `vmanomaly` with v1.15.0
- additional chapters of FAQ and model pages

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit b670fa1a29)
2024-08-07 16:59:21 +02:00
Anton L
79008b712f
app/vmselect/graphite: respect denyPartialResponse for graphite requests (#6748)
VM has different responses to equivalent queries for MetricsQL and
GraphiteQL in case of failed access to one of vmstorage node of the
cluster vmstorage nodes. For GraphiteQL, the denyPartialResponse feature
is not used, it is always true, which is not always correct (depending
on the configuration).

In the PR I have removed the hardcoded denyPartialResponse for
GraphiteQL, just like MetricsQL does.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 12:34:23 +02:00
Hui Wang
13a21a3ba0
app/vmagent/remotewrite: make -remoteWrite.streamAggr.ignoreFirstIntervals of array type (#6744)
Make `-remoteWrite.streamAggr.ignoreFirstIntervals` of array type so it could
 accept multiple values which can be applied to the corresponding`-remoteWrite.url`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8f5c26d788)
2024-08-07 09:57:49 +02:00
Hui Wang
71ac65996b
app/vmagent/remotewrite: fix -streamAggr.dropInputLabels behavior (#6743)
Fix `-streamAggr.dropInputLabels` behavior  when global deduplication is enabled without `-streamAggr.config`.
Previously, `-remoteWrite.streamAggr.dropInputLabels` is misapplied.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4863605469)
2024-08-07 09:57:49 +02:00
hagen1778
ab7863a654
docs: mention __sample_limit__ in sd_configs
Follow-up after 994796367b

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3a88553315)
2024-08-07 09:57:48 +02:00
Anzor
7e32daa63a
app/vmagent: read __sample_limit__ from labels (#6665) (#6666)
By introducing this feature, users will have the ability to customize
the sampleLimit parameter on a per-target basis, providing more
flexibility and control over the job execution behavior.

(cherry picked from commit 994796367b)
2024-08-07 09:57:48 +02:00
Andrii Chubatiuk
341d3a7f53
docs: fixed docs build (#6765)
### Describe Your Changes

fixed yaml header in a guide doc, that causes hugo build error

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 86b473c476)
2024-08-06 16:32:58 +02:00
Yury Molodov
b4aec9ee05
vmui/logs: add display top streams in the hits graph (#6647)
### Describe Your Changes

- Adds support for displaying the top 5 log streams in the hits graph,
grouping the remaining streams into an "other" label.
   #6545

- Adds options to customize the graph display with bar, line, stepped
line, and points views.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 04c2232e45)
2024-08-06 16:30:12 +02:00
Andrii Chubatiuk
c885f3e7dc
docs: updated docs titles and links (#6741)
The changes are based on SEO report and supposed to improve
ranking and indexation by search engines by using prompt and unique titles
and by updating unreachable links.

It also updates links to have a simplified form and replaces relative links with absolute links
according to https://docs.victoriametrics.com/#documentation

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 2e16732fdb)
2024-08-06 16:30:11 +02:00
Zakhar Bessarab
a3a0bafe76
app/vlinsert/elasticsearch: add fake response for logstash requests (#6742)
### Describe Your Changes

This is needed in order to support standard Elasticsearch output in
Logstash pipelines.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6660

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 58b6c54da2)
2024-08-06 16:30:11 +02:00
Tommy
f7a59dcddc
docs: fix typo in VictoriaLogs FAQ.md (#6755)
Update VictoriaLogs FQA section to replace VictoriaMetrics with
VictoriaLogs.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit f23650ccf9)
2024-08-06 16:30:11 +02:00
Hui Wang
9f84c4fdfa
vmalert: respect HTTP headers defined in notifier configuration file (#6762)
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit c1b54779a2)
2024-08-06 16:30:10 +02:00
hagen1778
6f448c6424
docs: data ingestion
* rm extra epmty lines
* rename images in according to https://docs.victoriametrics.com/#images-in-documentation

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit be0b892ce6)
2024-08-06 16:30:10 +02:00
Mathias Palmersheim
e65265d2ac
docs: add vmagent and alloy data ingestion docs (#6678)
Adds Prometheus Grafana Alloy and vmagent to the data ingestion
protocols. Grafana Agent was not added since it has been deprecated in
favor of alloy

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit a46d554f74)
2024-08-06 16:30:10 +02:00
Zhu Jiekun
e8af156655
docs: Added grafana version requirement for vm+vl datasorce (#6760)
### Describe Your Changes

https://victoriametrics.slack.com/archives/C05UNTPAEDN/p1722833182319299

Sometimes users may use the wrong (lower) version of Grafana when
setting up the VictoriaLogs datasource.

It would be good to document the requirements of Grafana versions to use
VictoriaMetrics/VictoriaLogs datasource.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 7551d799f7)
2024-08-06 16:30:09 +02:00
Ivan Yatskevich
f2dd045b68
docs: fix typo in quick start guide (#6757)
### Describe Your Changes

Fix a typo in docs.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 3c48e5662f)
2024-08-06 16:30:09 +02:00
Zakhar Bessarab
0b1def6e24
app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749)
### Describe Your Changes

This is useful for clients which validate InfluxDB is available before
data ingestion can be started.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6653

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 9877a5e7d5)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-05 09:45:32 +02:00
Dmytro Kozlov
fdad3e94f5
vmctl: add --backoff-retries, --backoff-factor, --backoff-min-duration global command-line flags (#6639)
### Describe Your Changes

Added `--vm-backoff-retries`, `--vm-backoff-factor`,
`--vm-backoff-min-duration` and `--vm-native-backoff-retries`,
`--vm-native-backoff-factor`, `--vm-native-backoff-min-duration`
command-line flags to the `vmctl` app. Those changes will help to
configure the retry backoff policy for different situations.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6622
### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 6f401daacb)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-03 19:34:03 +02:00
Yury Molodov
00b108ca04
vmui/logs: improve UI functionality (#6688)
* add a toggle button to the "Group" tab that allows users to expand or collapse all groups at once
* introduce the ability to select a key for grouping logs within the "Group" tab
* display the number of entries within each log group.
* move the Markdown toggle to the general settings panel in the upper left corner.

(cherry picked from commit e06a19d85f)
2024-08-02 15:58:07 +02:00
f41gh7
092ea42ba8
docs: mention v1.102.x LTS release line
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 14:04:21 +02:00
hagen1778
8c768c0df2
docs: rm reference to step number as it could change in future
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 14:04:21 +02:00
f41gh7
91d989d2d1
add new LTS release v1.102.x
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 11:15:25 +02:00
hagen1778
803c02e6f3
docs: mention step for re-building js static files
Changes to .ts files in vmui or vmui for logs require re-building
static files that will be included into compiled binary afterwards.

We don't update static files on each .ts change PR because it results
in too many changes and complicates review.

So we need to update these static files before the actual release.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 11:15:25 +02:00
hagen1778
94feee9f54
docs: use absolute links instead of relatives
See https://docs.victoriametrics.com/#documentation

```
Use absolute links. This simplifies moving docs between different files.
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-02 11:15:24 +02:00
Yury Molodov
a93ee27a85
vmui/logs: add fields for tenant configuration (#6661)
Added fields for configuring AccountID and ProjectID
#6631
2024-08-02 11:14:42 +02:00
f41gh7
35fbff3429
docs/CHANGELOG.md: cut v1.102.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-01 14:23:44 +02:00
f41gh7
3c8c45b41b
vendor: updates metricsql to v0.77.0 with bugfix
Fixes panic if incorrect metricsql expression passed to the prettifier API.

Prettify function had misleading panic for duration expression formatting. It expected all WITH templates to be already parsed.
But WITH expression expand was removed.

Bug was introduced at e712a49898
and present at v1.98.0+ releases

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6736
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-01 12:39:06 +02:00
hagen1778
0303765531
docs: follow-up after 58e667c895
* add Logo guidelines to GitHub Readme, as it may have higher chances to be viewed by users
* rm Logo guidelines from cluster version docs, as it makes no sense anymore for this page
* rm `picutre` tag from cluster version docs, as it is not used by GitHub anymore

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 77b768089f)
2024-07-31 16:20:28 +02:00
Andrii Chubatiuk
56a6e680e3
docs: grouped changelog docs, removed old make commands, replaced docs in root README with official docs links (#6727)
### Describe Your Changes

- replace docs in root README with a link to official documentation
- remove old make commands for documentation
- remove redundant "VictoriaMetrics" from document titles
- merge changelog docs into a section
- rm content of Single-server-VictoriaMetrics.md as it can be included from docs/README
- add basic information to README in the root folder, so it will be useful for github users
- rm `picture` tag from docs/README as it was needed for github only, we don't display VM logo at docs.victoriametrics.com
- update `## documentation` section in docs/README to reflect the changes
- rename DD pictures, as they now belong to docs/README

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 58e667c895)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-31 16:15:08 +02:00
Yury Molodov
7d37ca3159
vmui: fix auto-completion triggers (#6566)
### Describe Your Changes

- Fixes auto-complete triggers according to [these
comments](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5866#issuecomment-2065273421).
- Fixes loading and displaying suggestions when there is no metric in
the expression.
   Related issue: #6153
- Adds quotes when inserting label values.
   Related issue: #6260

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 53919327b2)
2024-07-31 16:09:18 +02:00
hagen1778
9b543a1394
docs: rm typo in vmalert docs
typo was added in b9f7c3169a

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b8b6c565f4)
2024-07-31 16:09:15 +02:00