Commit Graph

371 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
fe9e6b7495
docs/VictoriaLogs: replace incorrect usafe of VictoriaMetrics with VictoriaLogs
(cherry picked from commit 91f5417572)
2024-10-15 11:49:30 +02:00
Aliaksandr Valialkin
f7496c17e4
docs/VictoriaLogs/Roadmap.md: remove OpenTelemetry, Fluentd and Telegraf from the list of data ingestion protocols, which are going to be supported by VictoriaLogs
These protocols are already supported:

- OpenTelemetry protocol - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4839 ,
  https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6218 and 01430a155c

- Fluentd protocol - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7098
  and 05a64a8c14

- Telegraf protocol - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310 ,
  https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6443 and 1731c0eabf

(cherry picked from commit 200d723b9a)
2024-10-15 11:49:29 +02:00
Aliaksandr Valialkin
e581338b84
lib/logstorage: make sure that bs.br is non-nil before checking br.bs.bsw.bh.rowsCount there
br.bs may be nil when br contains the block with additional filters applied during pipe calculations.
For example, `* | count() if (error) errors`.

(cherry picked from commit 867f671cc4)
2024-10-15 11:49:29 +02:00
Zakhar Bessarab
4b1fe0d656
docs/victoria-logs: fix typo (#7245)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 2239f5829f)
2024-10-15 11:49:29 +02:00
Aliaksandr Valialkin
d2352fd84f
docs/VictoriaLogs/querying/vlogscli.md: add the list of features of vlogscli
(cherry picked from commit b9a8c1ff3a)
2024-10-11 14:28:07 +02:00
n4mine
e2f0fe8315
docs: fix typo in docs/VictoriaLogs/data-ingestion/Vector.md (#7222)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit f060b67da5)
2024-10-11 14:28:07 +02:00
Aliaksandr Valialkin
0fe533e0be
deployment: update VictoriaLogs from v0.34.0-victorialogs to v0.35.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.35.0-victorialogs

(cherry picked from commit baa87b5b36)
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
ff63816b06
docs/VictoriaLogs: cut v0.35.0 release
(cherry picked from commit 252aa792f7)
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
8d4c1cb1d8
docs/VictoriaLogs: make vlogscli more visible
(cherry picked from commit 9413b2de91)
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
4bb5f588bc
app/vlogscli: add -accountID and -projectID command-line flags for querying the given tenants
(cherry picked from commit ad5d8097da)
2024-10-11 14:27:45 +02:00
Aliaksandr Valialkin
d07e09b1e4
app/vlogscli: add support for live tailing
(cherry picked from commit e31625e0b2)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

# Conflicts:
#	Makefile
2024-10-11 14:27:26 +02:00
n4mine
fb47859c59
docs: fix typo in docs/VictoriaLogs/data-ingestion/Vector.md (#7208)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 344d61da79)
2024-10-11 14:25:22 +02:00
Aliaksandr Valialkin
00bd1c68a2
deployment: update VictoriaLogs image tag from v0.33.0-victorialogs to v0.34.0-victorialogs
Some checks failed
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.34.0-victorialogs
2024-10-08 12:30:49 +02:00
Aliaksandr Valialkin
db75455fbd
docs/VictoriaLogs/CHANGELOG.md: cut v0.34.0 release 2024-10-08 12:21:50 +02:00
Aliaksandr Valialkin
efe5935497
app/vlogscli: add ability to display query results in logfmt, single-line and multi-line json modes
(cherry picked from commit 492190885d)
2024-10-07 14:46:21 +02:00
Aliaksandr Valialkin
2e5dbd6f91
app/vlogscli: return back sorting result fields by name
This simplifies locating the needed field when the number of fields per each returned result is big

(cherry picked from commit daad96b3a5)
2024-10-07 14:46:20 +02:00
Aliaksandr Valialkin
026560df73
app/vlogscli: preserve the original order of fields in the displayed responses
Some checks failed
build / Build (push) Has been cancelled
CodeQL Go / Analyze (push) Has been cancelled
main / lint (push) Has been cancelled
main / test (test-full) (push) Has been cancelled
main / test (test-full-386) (push) Has been cancelled
main / test (test-pure) (push) Has been cancelled
2024-10-05 21:30:10 +02:00
Aliaksandr Valialkin
3809efc1ad
docs/VictoriaLogs/LogsQL.md: remove redundant replace word in examples for conditional replace
Thanks to @p5i for the bugreport at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7168
2024-10-05 21:30:06 +02:00
Aliaksandr Valialkin
7a44614e0b
lib/logstorage: add len pipe for calculating byte length of log field values
(cherry picked from commit 364f084b43)
2024-10-04 10:42:51 +02:00
Aliaksandr Valialkin
ff151f2caa
docs/VictoriaLogs/querying/vlogscli.md: typo fixes 2024-10-01 21:11:51 +02:00
Aliaksandr Valialkin
856c189688
docs/VictoriaLogs/querying/vlogscli.md: provide instructions on how to download run vlogscli 2024-10-01 16:13:35 +02:00
Aliaksandr Valialkin
ee784f9334
deployment: update VictoriaLogs docker image from v0.32.1-victorialogs to v0.33.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.33.0-victorialogs
2024-10-01 13:58:36 +02:00
Aliaksandr Valialkin
b2c3dbef09
docs/VictoriaLogs/CHANGELOG.md: cut v0.33.0-victorialogs release 2024-10-01 13:42:27 +02:00
Aliaksandr Valialkin
81f3e07e1e
lib/logstorage: do not count dictionary values which have no matching logs in count_uniq stats function
Create blockResultColumn.forEachDictValue* helper functions for visiting matching
dictionary values. These helper functions should prevent from counting dictionary values
without matching logs in the future.

This is a follow-up for 0c0f013a60
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7152
2024-10-01 13:36:27 +02:00
Aliaksandr Valialkin
8c55b699f4
app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:24:53 +02:00
Aliaksandr Valialkin
285a4dde11
docs/VictoriaLogs/FAQ.md: clarify answers a bit 2024-10-01 12:24:03 +02:00
Aliaksandr Valialkin
975f0f3610
deployment/docker: update VictoriaLogs from v0.32.0-victorialogs to v0.32.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.1-victorialogs
2024-09-30 14:38:27 +02:00
Aliaksandr Valialkin
c07746c1af
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.1-victorialogs release 2024-09-30 14:30:34 +02:00
Aliaksandr Valialkin
dbcf06cd85
lib/logstorage: skip values with zero hits for 'uniq', 'top' and 'field_values' pipes
See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/72#issuecomment-2352078483
2024-09-30 14:16:21 +02:00
Aliaksandr Valialkin
bf746f9975
deployment: update VictoriaLogs docker image from v0.31.0-victorialogs to v0.32.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.0-victorialogs
2024-09-29 14:54:42 +02:00
Aliaksandr Valialkin
3babcb0bbd
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.0-victorialogs 2024-09-29 14:48:36 +02:00
Aliaksandr Valialkin
58d1e517de
lib/logstorage: clear hits slice obtained from encoding.GetUint64s() before updating it with hits for valueTypeDict column
encoding.GetUint64s() returns uninitialized slice, which may contain arbitrary values.
So values in this slice must be reset to zero before using it for counting hits in `uniq` and `top` pipes.
2024-09-29 10:29:50 +02:00
Aliaksandr Valialkin
7f8b1300a9
lib/logstorage: add non-empty if (...) condition to automatically generated result names in stats pipe
This allows executing queries with `stats` pipe, which calculate multiple results with the same functions,
but with different `if (...)` conditions. For example:

  _time:5m | count(), count() if (error)

Previously such queries couldn't be executed becasue automatically generated name for the second result
didn't include `if (error)`, so names for both results were identical - `count(*)`.
2024-09-29 09:52:19 +02:00
Aliaksandr Valialkin
04c73d54d4
lib/logstorage: support order alias for sort pipe
Now the following queries are equivalents:

    _time:5s | sort by (_time)

    _time:5s | order by (_time)

This is needed for convenience, since `order by` is commonly used in other query languages such as SQL.
2024-09-29 09:52:18 +02:00
Aliaksandr Valialkin
0f1b3852dd
app/vlinsert: support unix timestamps in seconds and milliseconds in JSON stream data ingestion API 2024-09-28 21:57:19 +02:00
Aliaksandr Valialkin
b8fa213310
app/vlinsert: accept unix timestamp in seconds additionally to milliseconds at ElasticSearch bulk API
Timestamps in seconds are sometimes used for data ingestion via ElasticSearch bulk API
2024-09-28 21:21:19 +02:00
Andrii Chubatiuk
6adb3f0dc2
victorialogs: marked fluentd support in roadmap, added syslog example (#7098)
### Describe Your Changes

Marked fluentd in victorialogs roadmap
Added fluentd syslog example setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 05a64a8c14)
2024-09-27 14:43:04 +02:00
Aliaksandr Valialkin
7ef9600da9
deployment: update VictoriaLogs docker image from v0.30.1-victorialogs to v0.31.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.31.0-victorialogs
2024-09-27 13:59:49 +02:00
Aliaksandr Valialkin
8c62845211
docs/VictoriaLogs/CHANGELOG.md: cut v0.31.0-victorialogs release 2024-09-27 13:54:24 +02:00
Yury Molodov
64793ff5f0
vmui/logs: improve graph usability (#7025)
### Describe Your Changes

- Show the time range in the tooltip when hovering over staircase
graphs.
- Use bolder lines for staircase graphs.
- Increase the number of steps on the staircase graph to 100.
- Reduce the maximum width of the tooltip to 1/3 of the screen.
- Insert only the label name under the cursor into the query input field
when `Ctrl`-clicking the line legend.

See [this
comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545#issuecomment-2336805237).

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-27 13:20:18 +02:00
Aliaksandr Valialkin
1a6313ca68
lib/logstorage: allow using - instead of ! as a shorthand for NOT operator in LogsQL 2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
b60cb98377
lib/logstorage: support skipping _stream: prefix for stream filters
'_stream:{...}' can be written as '{...}'

This simplifies writing queries with stream filters, and makes them more familier to Loki users.
2024-09-27 13:15:55 +02:00
Yury Molodov
b95af2accf
vmui: add functionality to preserve selected columns (#7037)
### Describe Your Changes

1) Changed table settings from a popup to a modal window to simplify
future functionality additions.
2) Added functionality to save selected columns when data is modified or
the page is reloaded. See #7016.

<details>
  <summary>Example screenshots</summary>

<img alt="demo-1" width="600"
src="https://github.com/user-attachments/assets/a5d9a910-363c-4931-8b12-18ea8b3d97d8"/>

</details>

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit c896bf340d)
2024-09-27 12:40:52 +02:00
Aliaksandr Valialkin
fc6ddad117
deployment: update VictoriaLogs docker image tag from v0.30.0-victorialogs to v0.30.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.1-victorialogs
2024-09-27 11:26:31 +02:00
Aliaksandr Valialkin
ebb2c605f9
docs/VictoriaLogs/CHANGELOG.md: cut v0.30.1-victorialogs release 2024-09-27 11:21:28 +02:00
Aliaksandr Valialkin
bc0bb0c36a
lib/logstorage: consistently sort stream contexts belonging to different streams by the minimum time seen in the matching logs
This should simplify debugging of stream_context output, since it remains stable over repeated requests.
2024-09-27 11:21:28 +02:00
Aliaksandr Valialkin
e83dc8b30a
docs/VictoriaLogs/LogsQL.md: use proper heading for blocks_count pipe docs
All the links in docs assume that the heading for pipe docs should end with `pipe` word.
This fixes broken links.

This is a follow-up for e9950f6307
2024-09-27 09:46:08 +02:00
Aliaksandr Valialkin
f9102b9996
deployment: update VictoriaLogs docker image from v0.29.0-victorialogs to v0.30.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.0-victorialogs
2024-09-27 09:34:18 +02:00
Aliaksandr Valialkin
ce8eda4b51
docs/VictoriaLogs/CHANGELOG.md: cut v0.30.0-victorialogs release 2024-09-27 09:18:47 +02:00
Aliaksandr Valialkin
f5dfe1cacd
lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe
Previously only logs inside the selected time range could be returned by stream_context pipe.
For example, the following query could return up to 10 surrounding logs only for the last 5 minutes,
while most users expect this query should return up to 10 surrounding logs without restrictions on the time range.

    _time:5m panic | stream_context before 10

This enables the ability to implement stream context feature at VictoriaLogs web UI: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7063 .

Reduce memory usage when returning stream context over big log streams with millions of entries.
The new logic scans over all the log messages for the selected log stream, while keeping in memory only
the given number of surrounding logs. Previously all the logs for the given log stream on the selected time range
were loaded in memory before selecting the needed surrounding logs.
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6730 .

Reduce the scan performance for big log streams by fetching only the requested fields. For example, the following
query should be executed much faster than before if logs contain many fields other than _stream, _msg and _time:

    panic | stream_context after 30 | fields _stream, _msg, _time
2024-09-26 17:04:39 +02:00