jackyin
66789a8144
lib/logstorage: and filter results in unexpected response ( #6556 )
...
fix #6554
andfilter shouldn't return orfilter field which result in bloomfilter
return false.
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 975ed27a76
2024-09-03 10:49:25 +02:00
Aliaksandr Valialkin
a1decb5ca1
app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages
2024-07-10 03:05:55 +02:00
Aliaksandr Valialkin
b8a8d3d6f1
lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API
2024-07-10 00:39:16 +02:00
Aliaksandr Valialkin
d6415b2572
all: consistently use 'any' instead of 'interface{}'
...
'any' type is supported starting from Go1.18. Let's consistently use it
instead of 'interface{}' type across the code base, since `any` is easier to read than 'interface{}'.
2024-07-10 00:23:26 +02:00
Aliaksandr Valialkin
9edeecabc8
lib: consistently use f-tests instead of table-driven tests
...
This makes easier to read and debug these tests. This also reduces test lines count by 15% from 3K to 2.5K .
See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e .
While at it, consistently use t.Fatal* instead of t.Error*, since t.Error* usually leads
to more complicated and fragile tests, while it doesn't bring any practical benefits over t.Fatal*.
2024-07-09 22:39:13 +02:00
Aliaksandr Valialkin
6397c38a0a
lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings
...
The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .
See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-05 01:22:49 +02:00
Aliaksandr Valialkin
7426b40250
lib/logstorage: allow writing after N in front of before N at stream_context pipe
2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin
208a624d4d
lib/logstorage: properly search for the surrounding logs in stream_context pipe
...
The set of log fields in the found logs may differ from the set of log fields present in the log stream.
So compare only the log fields in the found logs when searching for the matching log entry in the log stream.
While at it, return _stream field in the delimiter log entry, since this field is used by VictoriaLogs Web UI
for grouping logs by log streams.
2024-07-01 02:33:00 +02:00
Aliaksandr Valialkin
76a58ae08d
lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax
2024-07-01 01:46:03 +02:00
Aliaksandr Valialkin
d0dca7b8c5
lib/logstorage: add delimiter between log chunks returned from | stream_context pipe
2024-07-01 01:46:02 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs
2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
2f28819bb1
lib/logstorage: it is safe using | unroll pipe in live tailing
...
`| unroll` pipe can make multiple copies of rows from the input row.
This doesn't break live tailing, so allow `| unroll` pipe in live tailing.
2024-06-27 19:45:12 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress
2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage
2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data
2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
...
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.
While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
Aliaksandr Valialkin
2a7fcba330
lib/logstorage: make golangci-lint happy
2024-06-25 03:06:28 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress
2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress
2024-06-20 03:10:37 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports
2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress
2024-06-17 12:13:25 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress
2024-06-11 17:51:01 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress
2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress
2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
a200fb433a
lib/logstorage: allow using eval keyword instead of math keyword in math pipe
2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress
2024-06-05 03:18:25 +02:00
Aliaksandr Valialkin
1ce8a9a751
lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc
2024-06-04 02:29:18 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress
2024-06-04 01:50:55 +02:00
Aliaksandr Valialkin
540bbb63a2
lib/logstorage: work-in-progress
2024-05-30 16:19:36 +02:00
Aliaksandr Valialkin
e83fd4a117
lib/logstorage: work-in-progress
2024-05-29 01:52:34 +02:00
Aliaksandr Valialkin
79c03fc35f
lib/logstorage: work-in-progress
2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin
ce5e4c842a
lib/logstorage: fix golangci-lint warnings
2024-05-26 02:02:41 +02:00
Aliaksandr Valialkin
afa597ce2a
lib/logstorage: work-in-progress
2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin
6427b3c3c0
lib/logstorage: work-in-progress
2024-05-25 22:59:21 +02:00
Aliaksandr Valialkin
9edbeca46b
lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll
2024-05-25 22:13:44 +02:00
Aliaksandr Valialkin
03fe4c8963
lib/logstorage: work-in-progress
2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress
2024-05-25 00:31:55 +02:00
Aliaksandr Valialkin
7a2a2f173e
lib/logstorage: work-in-progress
2024-05-24 03:07:07 +02:00
Alexander Marshalov
0b70c4c1f1
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:54:50 +02:00
Aliaksandr Valialkin
04d0dd2542
lib/logstorage: work-in-progress
2024-05-22 21:01:28 +02:00
Aliaksandr Valialkin
45fbcc74e0
lib/logstorage: fix golangci-lint warnings
2024-05-20 11:04:37 +02:00
Aliaksandr Valialkin
582e7d5439
lib/logstorage: work-in-progress
2024-05-20 04:09:15 +02:00
Aliaksandr Valialkin
28626db066
lib/logstorage: work-in-progress
...
(cherry picked from commit 0aa19a2837
2024-05-16 09:35:55 +02:00
Aliaksandr Valialkin
b1ee7bca1a
lib/logstorage: work-in-progress
2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin
f52275bbd7
lib/logstorage: work-in-progress
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258
2024-05-14 01:49:58 +02:00
Aliaksandr Valialkin
32193b6059
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit
...
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.
This improves performance of these functions in hot loops of VictoriaLogs a bit.
2024-05-14 01:30:25 +02:00
hagen1778
84a896cd6e
lib/logstorage: make linter happy
...
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 17283fab6c
2024-05-13 16:49:37 +02:00
Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin
87338633b1
lib/slicesutil: add helper functions for setting slice length and extending its capacity
...
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
2024-05-12 11:33:49 +02:00
wanshuangcheng
52a4ae0b28
chore: fix function names in comment ( #6076 )
...
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 15:38:51 +02:00
