Commit Graph

3327 Commits

Author SHA1 Message Date
Hui Wang
085bc1f15c
vmui: increase max query tab from 4 to 10 (#6546)
(cherry picked from commit 9da78f1e0e)
2024-07-01 16:40:42 +02:00
Hui Wang
87cb132f53
app/vmselect/netstorage: do not retry request when complexity limit i… (#6469)
…s already exceeded

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-01 16:38:15 +02:00
Andrii Chubatiuk
937ae2ca90
lib/streamaggr: added stale samples metric, added metrics labels (#6462)
### Describe Your Changes

- added stale metrics counters for input and output samples
- added labels for aggregator metrics =>
`name="{rwctx}:{aggrId}:{aggrSuffix}"`
   - rwctx - global or number starting from 1
   - aggrid - aggregator id starting from 1
   - aggrSuffix - <interval>_(by|without)_label1_label2_labeln
   e.g: `name="global:1:1m_without_instance_pod"`

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 861852f262)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-01 15:01:49 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
c9fc8079c4
app/vlinsert/syslog: properly skip empty lines in Syslog protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548
2024-06-28 14:09:45 +02:00
Aliaksandr Valialkin
bb6424aeca
app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint
This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545
in order to return top N groups with the biggest number of hits.
2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
b26acec9a8
app/vlselect: properly return live tailing results 2024-06-27 15:06:15 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
Andrii Chubatiuk
f79df2aa8b
app/vmauth: allow dropping host header (#6525)
### Describe Your Changes

Fixes #6453

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
Yury Molodov
6bde0196d8
vmui/logs: fix the update of the relative time range (#6517)
### Describe Your Changes

- Fixed the update of the relative time range when `Execute Query` is
clicked
- Optimized server requests: now, if an error occurs in the `/query`
request, the `/hits` request will not be executed.

#6345 (duplicates: #6440, #6312)

(cherry picked from commit 43342745ac)
2024-06-26 11:26:08 +02:00
Yury Molodov
904ec020ed
vmui: fix input cursor position reset (#6530)
### Describe Your Changes

This PR addresses the issue where the cursor jumps to the end of the
input fields in the modal settings window after each keystroke.

### Before fix:

![ezgif-7-4c69805cea](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/2e99e833-09e3-4b44-89aa-fc1bd3c4346d)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit e9b71a2883)
2024-06-26 11:25:47 +02:00
Yury Molodov
25f3e700a6
vmui: update package-lock.json (#6532)
1. Updated `package-lock.json` to resolve [Dependabot
alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/security/dependabot).
2. Updated types to align with the latest `Preact` update.

(cherry picked from commit 6cab811134)
2024-06-26 11:25:45 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
Aliaksandr Valialkin
30d1f0711f
app/vmselect/netstorage: add a comment explaining why all the samples in block are taken into account when checking the -search.maxSamplesPerQuery limit
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

This is a follow-up for b07a02c516
2024-06-25 03:06:42 +02:00
Aliaksandr Valialkin
a5445e09c2
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 5ecf439078.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin
f8ff09cd8d
app/vmui: run make vmui-update after 65f414acee 2024-06-24 23:21:19 +02:00
Aliaksandr Valialkin
faed1394d9
app/vmctl/prometheus/prometheus.go: add missing arg to tsdb.OpenDBReadOnly() function after updating github.com/prometheus/prometheus dependency from v0.52.1 to v0.53.0 in 5c55722db4
See c5a1cc9148
2024-06-24 23:16:30 +02:00
Andrii Chubatiuk
516848783e
deployment: build image for vmagent streamaggr benchmark (#6515)
### Describe Your Changes

optionally build vmagent image for benchmark
needed for https://github.com/VictoriaMetrics/ops/pull/1297

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 6b128da811)
2024-06-24 16:29:14 +02:00
hagen1778
1aadf2b267
app/vmalert: fix typo in replay error handling
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 279815818c)
2024-06-20 15:15:59 +02:00
hagen1778
d98577ae37
app/vmalert: follow-up bc37b279aa
* rm extra interface method for rw Client, as it has low applicability
and doesn't fit multitenancy well
* add `GetDroppedRows` method instead

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4ef76eed7b)
2024-06-20 15:15:58 +02:00
Hui Wang
a393b993d6
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512

(cherry picked from commit bc37b279aa)
2024-06-20 14:00:19 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
Yury Molodov
5e8e89f22d
vmui/logs: update footer links (#6498)
### Describe Your Changes
Update the links in the footer for logs:
[LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) and
[Documentation](https://docs.victoriametrics.com/victorialogs/)

(cherry picked from commit 13e3bb88a9)
2024-06-18 15:29:14 +02:00
Yury Molodov
88650abf97
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.

(cherry picked from commit 32fbffedd9)
2024-06-18 15:28:56 +02:00
Hui Wang
5be2f2c4e4
vmalert-tool: support file path with hierarchical patterns and regexp… (#6501)
…es, and http url in unittest cmd-line flag `-files`

(cherry picked from commit 3b8970802e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:17:42 +02:00
Aliaksandr Valialkin
ee114ca59c
app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint
This has been broken in 2b6a634ec0
2024-06-18 00:24:11 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
f74e6b0674
app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425 2024-06-17 22:30:28 +02:00
jackyin
4a6bf7f218
app/vmui: copy button shows undefined (#6495)
### Describe Your Changes
fix #6421
some aggregation func don't return \_\_name\_\_ value
2024-06-17 22:30:28 +02:00
Roman Khavronenko
df7e300071
app/vmselect/promql: check for ranged vectors in aggr funcs if implicit conversions are disabled (#6450)
Check for ranged vector arguments in aggregate expressions when
`-search.disableImplicitConversion` or `-search.logImplicitConversion`
are enabled.
 For example, `sum(up[5m])` will fail to execute if these flags are set.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [*] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6149adbe10)
2024-06-17 14:25:43 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
Hui Wang
5ecf439078
app/vmselect: fix the way of counting raw samples in single query (#6464)
The limit is specified with command-line flag
`-search.maxSamplesPerQuery`.
Previously, samples might be over-counted and query can't be fixed by
reducing time range.
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

(cherry picked from commit 6e395048d3)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 16:24:31 +02:00
jackyin
f69495cd5f
app/vmalert: fix VMAlert oauth2 error (#6478)
Properly set ClientSecret param for notifier.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5223981fed)
2024-06-14 15:21:30 +02:00
Andrii Chubatiuk
779436bd9c
app/vmalert: fixed path prefixes for system routes (#6435)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit eea361defb)
2024-06-14 14:14:54 +02:00
LHHDZ
41e4135371
app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401)
This change fixes the following panic:
```
2024-06-04T11:16:52.899Z        warn    app/vmauth/auth_config.go:353   cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally
panic: runtime error: integer divide by zero

goroutine 9 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58
panic({0x103115100?, 0x10338d700?})
        /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124
main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210
main.(*URLPrefix).getBackendURL(0x140000aa080)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8
```

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-06-12 11:47:44 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress 2024-06-11 17:51:01 +02:00
Nikolay
66fbea70a5
follow-up after 77f22fdb8d (#6458)
* fixes linter error
* simplify code a bit
* fixes bug with incorrectly set configSuccess metric. It was not set to
1 in case of config rollback

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-06-11 12:08:00 +02:00
noodles2hg
77f22fdb8d
[cluster/vminsert]:add reload -relabelConfig on the request to /-/reload (#3923)
When I use vminsert's `relabelConfig`, I found that now there is no
reloaded api. However, `vminsert` under `VM-Single` has it. So, I hope
to add it to the `cluster/vminster`.

---------

Signed-off-by: z-anshun <1179798460@qq.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-06-10 19:36:41 +02:00
Yury Molodov
2300e30ff3
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:39:25 +02:00
hagen1778
fdf0a936f0
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8d95522529)
2024-06-10 14:05:58 +02:00
Nikolay
0ce7f38e1c
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d44058bcd6)
2024-06-10 12:41:51 +02:00
Dmytro Kozlov
a4bdc14bc5
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 8b46bb0c41)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:44 +02:00
Hui Wang
028a80613f
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329,
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 61dce6f2a1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:29 +02:00
Aliaksandr Valialkin
3492f4e1fe
app/vmselect/vmui: run make vmui-update after c236e3c03c1bf8ca00292b800a839fcb300e7e51 and 04744c274c269f6b6efb45f68df11abe0fb0ce25 2024-06-07 16:39:06 +02:00
Aliaksandr Valialkin
9dfc7190fe
app/vlselect/vmui: run make vmui-logs-update after a68c2c0f17
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407
2024-06-06 12:21:47 +02:00
Yury Molodov
8cf417e1c7
vmui/logs: improve log display for group view (#6419)
### Describe Your Changes

1) Set the default limit to `50`. 
    #6408
2) Configure the default search to cover the `last 5 minutes` and
include all messages (`*`).
     #6405
3) In the header, display only streams and group by stream.
     #6406
4) Add log processing, without the fields `msg`, `time`, and `stream`.
5) When clicking on logs, display a list of all fields.
     #6407

<img width="400" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/666dcaa3-20fb-4828-b77b-1d849dd9a8ed">

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-06 12:21:46 +02:00
Dima Lazerka
362ee240cd
vmui: Improve DownloadConfig button interaction with VMAnomaly (#6397)
Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
2024-06-06 12:12:58 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00