Aliaksandr Valialkin
e5537bc64d
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
a98fb495c6
lib/logstorage: allow specifying _time filter offset without time range
...
This is useful when builiding graphs on time ranges in the past.
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
66b2987f49
lib/logstorage: optimize query imeediately after its parsing
...
This eliminates possible bugs related to forgotten Query.Optimize() calls.
This also allows removing optimize() function from pipe interface.
While at it, drop filterNoop inside filterAnd.
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
0550093802
app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint.
...
This reduces the probability of forgotten call to Query.Optimize().
2024-11-08 16:43:53 +01:00
Aliaksandr Valialkin
5ed54ebadf
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
2024-11-06 18:55:45 +01:00
Aliaksandr Valialkin
7603446850
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
...
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365
2024-10-30 17:59:49 +01:00
Hui Wang
68bad22fd2
vmalert: integrate with victorialogs ( #7255 )
...
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706 .
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md .
Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254 .
Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling )
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289 )
for details.
Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-29 16:30:39 +01:00
Aliaksandr Valialkin
508e498ae3
lib/logstorage: follow-up for 72941eac36
...
- Allow dropping metrics if the query result contains at least a single metric.
- Allow copying by(...) fields.
- Disallow overriding by(...) fields via `math` pipe.
- Allow using `format` pipe in stats query. This is useful for constructing some labels from the existing by(...) fields.
- Add more tests.
- Remove the check for time range in the query filter according to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254/files#r1803405826
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254
2024-10-16 19:43:52 +02:00
Hui Wang
72941eac36
victorialogs: add more checks for stats query APIs ( #7254 )
...
1. Verify if field in [fields
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe )
exists. If not, it generates a metric with illegal float value "" for
prometheus metrics protocol.
2. check if multiple time range filters produce conflicted query time
range, for instance:
```
query: _time: 5m | stats count(),
start:2024-10-08T10:00:00.806Z,
end: 2024-10-08T12:00:00.806Z,
time: 2024-10-10T10:02:59.806Z
```
must give no result due to invalid final time range.
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-16 19:25:43 +02:00
Aliaksandr Valialkin
7b475ed95d
lib/logstorage: disallow using pipe names as the first unquoted words in filter pipe
...
Improperly written pipes could be silently parsed as filter pipe.
For example, the following query:
* | by (x)
was silently parsed to:
* | filter "by" x
It is better to return error, so the user could identify and fix invalid pipe
instead of silently executing invalid query with `filter` pipe.
2024-10-09 16:10:13 +02:00
Aliaksandr Valialkin
6acf543b90
lib/logstorage: disallow using by as the first word in log filters, since it frequently clashes with stats by(...) pipe where stats word is omitted
2024-10-09 15:53:15 +02:00
Aliaksandr Valialkin
89686094a0
lib/logstorage: allow special chars in unquoted _stream tag names and values
...
build / Build (push) Waiting to run
CodeQL Go / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
This simplifies writing _stream filters. For example,
{foo-bar=abc:de}
can be written instead of
{"foo-bar"="abc:de"}
2024-10-07 15:10:03 +02:00
Aliaksandr Valialkin
630211cfed
app/vlogscli: add interactive command-line tool for querying VictoriaLogs
2024-10-01 12:23:07 +02:00
Aliaksandr Valialkin
1da4650143
lib/logstorage: allow using ! in unescaped phrase
...
Previously the phrase filter with `!` was treated unexpectedly.
For example, `foo!bar` filter was treated at `foo AND NOT bar`,
while most users expect that it matches "foo!bar" phrase.
This commit aligns with users' expectations.
2024-09-29 11:14:15 +02:00
Aliaksandr Valialkin
60183c7c79
lib/logstorage: allow using - instead of ! in front of (...)
2024-09-29 11:12:22 +02:00
Aliaksandr Valialkin
09b309a82e
lib/logstorage: allow using - instead of ! as a shorthand for NOT operator in LogsQL
2024-09-27 13:14:47 +02:00
Aliaksandr Valialkin
76c1b0b8ea
lib/logstorage: support skipping _stream: prefix for stream filters
...
'_stream:{...}' can be written as '{...}'
This simplifies writing queries with stream filters, and makes them more familier to Loki users.
2024-09-27 13:14:46 +02:00
Aliaksandr Valialkin
255d1d4e13
app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop
...
Previously the original timestamp was used in the copied query, so _time:duration filters
were applied to the original time range: (timestamp-duration ... timestamp]. This resulted
in stopped live tailing, since new logs have timestamps bigger than the original time range.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7028
2024-09-26 08:57:23 +02:00
Aliaksandr Valialkin
e9950f6307
lib/logstorage: add blocks_count pipe
...
This pipe is useful for debugging purposes when the number of processed blocks must be calculated for the given query:
<query> | blocks_count
This helps detecting the root cause of query performance slowdown in cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070
2024-09-25 19:17:48 +02:00
Aliaksandr Valialkin
657988ac3a
app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg
...
Previously the query could return incorrect results, since the query timestamp was updated with every Query.Clone() call
during iterative search for the time range with up to limit=N rows.
While at it, optimize queries, which find low number of matching logs, while spend a lot of CPU time for searching
across big number of logs. The optimization reduces the upper bound of the time range to search if the current time range
contains zero matching rows.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785
2024-09-08 14:32:23 +02:00
Aliaksandr Valialkin
eaee2d7db4
lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions
2024-09-08 11:24:44 +02:00
Aliaksandr Valialkin
0a40064a6f
app/vlselect: add /select/logsql/stats_query_range endpoint for building time series panels in VictoriaLogs plugin for Grafana
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6943
Updates https://github.com/VictoriaMetrics/victorialogs-datasource/issues/61
2024-09-07 00:41:47 +02:00
Aliaksandr Valialkin
c9bb4ddeed
app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706
2024-09-06 23:06:43 +02:00
Aliaksandr Valialkin
0205170409
lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant
2024-09-06 16:17:04 +02:00
Aliaksandr Valialkin
aa9bb99527
lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API
2024-07-10 00:39:28 +02:00
Aliaksandr Valialkin
d4ca651547
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs
2024-06-28 19:14:29 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress
2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin
de7450b7e0
lib/logstorage: work-in-progress
2024-06-24 23:27:12 +02:00
Aliaksandr Valialkin
7229dd8c33
lib/logstorage: work-in-progress
2024-06-20 03:10:08 +02:00
Aliaksandr Valialkin
2b6a634ec0
lib/logstorage: work-in-progress
2024-06-17 12:13:18 +02:00
Aliaksandr Valialkin
8f5dc966f6
lib/logstorage: work-in-progress
2024-06-11 17:50:32 +02:00
Aliaksandr Valialkin
0521e58a09
lib/logstorage: work-in-progress
2024-06-10 18:42:19 +02:00
Aliaksandr Valialkin
55d8379ae6
lib/logstorage: work-in-progress
2024-06-06 12:27:05 +02:00
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress
2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin
539fce9227
lib/logstorage: work-in-progress
2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin
0aafca29be
lib/logstorage: work-in-progress
2024-05-28 19:29:41 +02:00
Aliaksandr Valialkin
1e203f35f7
lib/logstorage: work-in-progress
2024-05-26 01:55:21 +02:00
Aliaksandr Valialkin
dc55146752
lib/logstorage: work-in-progress
2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin
e2590f0485
lib/logstorage: work-in-progress
2024-05-25 00:30:58 +02:00
Aliaksandr Valialkin
4b458370c1
lib/logstorage: work-in-progress
2024-05-24 03:06:55 +02:00
Alexander Marshalov
7da541360e
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:46:50 +02:00
Aliaksandr Valialkin
22107421eb
lib/logstorage: work-in-progress
2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin
ad505a7a9a
lib/logstorage: work-in-progress
2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin
0aa19a2837
lib/logstorage: work-in-progress
2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin
da3af090c6
lib/logstorage: work-in-progress
2024-05-14 03:05:03 +02:00
Aliaksandr Valialkin
9dbd0f9085
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:33:31 +02:00
XLONG96
a5795f533d
lib/logstorage: avoid panic when parsing regex with stream filter ( #5897 )
2024-02-29 15:31:54 +02:00
Aliaksandr Valialkin
a8000b74c5
lib/logstorage: properly encode "offset" search word just after _time filter
2023-07-18 16:00:06 -07:00
Aliaksandr Valialkin
ed00b03ecb
lib/logstorage: add abilty to speficy offset for the selected _time filter
...
The following syntax is supported: _time:filter offset off
For example:
- _time:5m offset 1h - 5-minute duration one hour before the current time
- _time:2023 offset 2w - 2023 year with the 2 weeks offset in the past
2023-07-17 19:07:42 -07:00
Aliaksandr Valialkin
bdfb80668d
lib/logstorage: support for short form of _time:(now-duration, now] filter: _time:duration
2023-07-17 19:07:40 -07:00
