Commit Graph

3194 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
7c8c040502
app/vlselect: properly return live tailing results 2024-06-27 15:05:57 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
Andrii Chubatiuk
e666d64f1d
app/vmauth: allow dropping host header (#6525)
### Describe Your Changes

Fixes #6453

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 17:42:57 +02:00
Yury Molodov
43342745ac
vmui/logs: fix the update of the relative time range (#6517)
### Describe Your Changes

- Fixed the update of the relative time range when `Execute Query` is
clicked
- Optimized server requests: now, if an error occurs in the `/query`
request, the `/hits` request will not be executed.

#6345 (duplicates: #6440, #6312)
2024-06-26 11:23:22 +02:00
Yury Molodov
e9b71a2883
vmui: fix input cursor position reset (#6530)
### Describe Your Changes

This PR addresses the issue where the cursor jumps to the end of the
input fields in the modal settings window after each keystroke.

### Before fix:

![ezgif-7-4c69805cea](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/2e99e833-09e3-4b44-89aa-fc1bd3c4346d)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 11:14:12 +02:00
Yury Molodov
6cab811134
vmui: update package-lock.json (#6532)
1. Updated `package-lock.json` to resolve [Dependabot
alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/security/dependabot).
2. Updated types to align with the latest `Preact` update.
2024-06-26 11:11:59 +02:00
Aliaksandr Valialkin
dff5008392
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:33 +02:00
Aliaksandr Valialkin
3eacd43fff
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
Aliaksandr Valialkin
9e1c037249
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:53:38 +02:00
Aliaksandr Valialkin
6a0cf2cd29
app/vmselect/netstorage: add a comment explaining why all the samples in block are taken into account when checking the -search.maxSamplesPerQuery limit
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

This is a follow-up for b07a02c516
2024-06-25 03:01:43 +02:00
Aliaksandr Valialkin
b07a02c516
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 6e395048d3.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:43:57 +02:00
Aliaksandr Valialkin
d91125b604
app/vmui: run make vmui-update after 65f414acee 2024-06-24 23:20:33 +02:00
Aliaksandr Valialkin
4dd5fe895e
app/vmctl/prometheus/prometheus.go: add missing arg to tsdb.OpenDBReadOnly() function after updating github.com/prometheus/prometheus dependency from v0.52.1 to v0.53.0 in 5c55722db4
See c5a1cc9148
2024-06-24 23:15:56 +02:00
Andrii Chubatiuk
6b128da811
deployment: build image for vmagent streamaggr benchmark (#6515)
### Describe Your Changes

optionally build vmagent image for benchmark
needed for https://github.com/VictoriaMetrics/ops/pull/1297

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-24 16:28:50 +02:00
hagen1778
279815818c
app/vmalert: fix typo in replay error handling
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-20 15:15:34 +02:00
hagen1778
4ef76eed7b
app/vmalert: follow-up bc37b279aa
* rm extra interface method for rw Client, as it has low applicability
and doesn't fit multitenancy well
* add `GetDroppedRows` method instead

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-20 15:12:53 +02:00
Hui Wang
bc37b279aa
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512
2024-06-20 13:20:40 +02:00
Hui Wang
fb7454a14d
vmalert-tool: exit immediately when rule group execute failed (#6509)
g.ExecOnce() shouldn't be failed at all. If it fails, it might be bug or
something wrong with tmp vm datasource, exit immediately.
2024-06-20 11:47:00 +02:00
Aliaksandr Valialkin
7229dd8c33
lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
Yury Molodov
13e3bb88a9
vmui/logs: update footer links (#6498)
### Describe Your Changes
Update the links in the footer for logs:
[LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) and
[Documentation](https://docs.victoriametrics.com/victorialogs/)
2024-06-18 15:25:32 +02:00
Yury Molodov
32fbffedd9
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.
2024-06-18 15:23:21 +02:00
Hui Wang
3b8970802e
vmalert-tool: support file path with hierarchical patterns and regexp… (#6501)
…es, and http url in unittest cmd-line flag `-files`
2024-06-18 14:14:30 +02:00
hagen1778
ede9004850
app/vmalert-tool: typo fix
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:05:36 +02:00
Hui Wang
d62f303e53
vmalert-tool: exit normally when no rule is defined under rule group (#6502)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 14:00:06 +02:00
Aliaksandr Valialkin
3eda4617c0
app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint
This has been broken in 2b6a634ec0
2024-06-18 00:23:25 +02:00
Aliaksandr Valialkin
e498fa6960
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:34 +02:00
Aliaksandr Valialkin
478468e6cd
app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425 2024-06-17 22:28:26 +02:00
jackyin
65f414acee
app/vmui: copy button shows undefined (#6495)
### Describe Your Changes
fix #6421
some aggregation func don't return \_\_name\_\_ value
2024-06-17 16:02:00 +02:00
Roman Khavronenko
6149adbe10
app/vmselect/promql: check for ranged vectors in aggr funcs if implicit conversions are disabled (#6450)
Check for ranged vector arguments in aggregate expressions when
`-search.disableImplicitConversion` or `-search.logImplicitConversion`
are enabled.
 For example, `sum(up[5m])` will fail to execute if these flags are set.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [*] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-17 14:21:16 +02:00
Aliaksandr Valialkin
2b6a634ec0
lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
Hui Wang
6e395048d3
app/vmselect: fix the way of counting raw samples in single query (#6464)
The limit is specified with command-line flag
`-search.maxSamplesPerQuery`.
Previously, samples might be over-counted and query can't be fixed by
reducing time range.
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
2024-06-14 15:40:30 +02:00
jackyin
5223981fed
app/vmalert: fix VMAlert oauth2 error (#6478)
Properly set ClientSecret param for notifier.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 15:06:14 +02:00
Andrii Chubatiuk
eea361defb
app/vmalert: fixed path prefixes for system routes (#6435)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-06-14 13:34:23 +02:00
LHHDZ
3a45bbb4e0
app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401)
This change fixes the following panic:
```
2024-06-04T11:16:52.899Z        warn    app/vmauth/auth_config.go:353   cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally
panic: runtime error: integer divide by zero

goroutine 9 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58
panic({0x103115100?, 0x10338d700?})
        /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124
main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210
main.(*URLPrefix).getBackendURL(0x140000aa080)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8
```

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-06-12 12:30:44 +02:00
Aliaksandr Valialkin
8f5dc966f6
lib/logstorage: work-in-progress 2024-06-11 17:50:32 +02:00
Yury Molodov
84088e5a2d
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:38:13 +02:00
hagen1778
8d95522529
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 14:02:46 +02:00
Nikolay
d44058bcd6
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:36:37 +02:00
Dmytro Kozlov
8b46bb0c41
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:20:52 +02:00
Hui Wang
61dce6f2a1
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329, 
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:09:47 +02:00
Aliaksandr Valialkin
e8ab8944e6
app/vmselect/vmui: run make vmui-update after c236e3c03c1bf8ca00292b800a839fcb300e7e51 and 04744c274c269f6b6efb45f68df11abe0fb0ce25 2024-06-07 16:39:19 +02:00
Aliaksandr Valialkin
21fafd550c
app/vlselect/vmui: run make vmui-logs-update after a68c2c0f17
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407
2024-06-06 12:18:08 +02:00
Yury Molodov
a68c2c0f17
vmui/logs: improve log display for group view (#6419)
### Describe Your Changes

1) Set the default limit to `50`. 
    #6408
2) Configure the default search to cover the `last 5 minutes` and
include all messages (`*`).
     #6405
3) In the header, display only streams and group by stream.
     #6406
4) Add log processing, without the fields `msg`, `time`, and `stream`.
5) When clicking on logs, display a list of all fields.
     #6407

<img width="400" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/666dcaa3-20fb-4828-b77b-1d849dd9a8ed">

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-06 12:14:06 +02:00
Dima Lazerka
c57c16925d
vmui: Improve DownloadConfig button interaction with VMAnomaly (#6397)
Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
2024-06-06 11:07:59 +02:00
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin
539fce9227
lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
hagen1778
a5f81f67fd
app/vmalert: rm extra response for unsupported path
Unsupported path is already handled by `lib/httpserver`.
This prevents from misleading errors in logs caused by double-writing response headers.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 12:52:02 +02:00
Hui Wang
e3e40cb848
vmalert-tool: fix float values template in input_series (#6395)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6391

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:49:44 +02:00
hagen1778
6d8e02f278
chore: follow-up after c740a8042e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 10:26:57 +02:00
Nikolay
b97916276f
app/vmalert: adds idleConnTimeout flags and retry trivial network errors (#6382)
* "*.idleConnTimeout" flags must reduce probability of `write: broken
pipe` and `read: connection reset by peer` errors Those errors may occur
if remote server closes TCP socket for connection, while it's still
exist at client.
* single time retries for `write: broken pipe` and `read: connection
reset by peer` must handle a case for incorrectly configured timeouts at
middleware proxies, mitigate minor network issues.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-05-30 17:54:42 +02:00