86 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	bb0deb7ac4	lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax	2024-07-01 01:44:17 +02:00
Aliaksandr Valialkin	dc291d8980	lib/logstorage: add delimiter between log chunks returned from | stream_context pipe	2024-07-01 01:30:37 +02:00
Aliaksandr Valialkin	d4ca651547	lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs	2024-06-28 19:14:29 +02:00
Aliaksandr Valialkin	0730f1324d	lib/logstorage: it is safe using | unroll pipe in live tailing ... `| unroll` pipe can make multiple copies of rows from the input row. This doesn't break live tailing, so allow `| unroll` pipe in live tailing.	2024-06-27 19:44:57 +02:00
Aliaksandr Valialkin	87f1c8bd6c	lib/logstorage: work-in-progress	2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin	dff5008392	app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage	2024-06-25 17:30:33 +02:00
Aliaksandr Valialkin	3eacd43fff	lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data	2024-06-25 14:53:39 +02:00
Aliaksandr Valialkin	9e1c037249	lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() ... The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets. While at it, make tryParseTimestampISO8601 function private in order to prevent from improper usage of this function from outside the lib/logstorage package. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508	2024-06-25 14:53:38 +02:00
Aliaksandr Valialkin	7252c5d258	lib/logstorage: make golangci-lint happy	2024-06-25 03:04:21 +02:00
Aliaksandr Valialkin	de7450b7e0	lib/logstorage: work-in-progress	2024-06-24 23:27:12 +02:00
Aliaksandr Valialkin	7229dd8c33	lib/logstorage: work-in-progress	2024-06-20 03:10:08 +02:00
Aliaksandr Valialkin	e498fa6960	app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports	2024-06-17 23:16:34 +02:00
Aliaksandr Valialkin	2b6a634ec0	lib/logstorage: work-in-progress	2024-06-17 12:13:18 +02:00
Aliaksandr Valialkin	8f5dc966f6	lib/logstorage: work-in-progress	2024-06-11 17:50:32 +02:00
Aliaksandr Valialkin	0521e58a09	lib/logstorage: work-in-progress	2024-06-10 18:42:19 +02:00
Aliaksandr Valialkin	55d8379ae6	lib/logstorage: work-in-progress	2024-06-06 12:27:05 +02:00
Aliaksandr Valialkin	80a7c65ab7	lib/logstorage: allow using eval keyword instead of math keyword in math pipe	2024-06-05 10:07:49 +02:00
Aliaksandr Valialkin	43cf221681	lib/logstorage: work-in-progress	2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin	96c29ab403	lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc	2024-06-04 02:29:10 +02:00
Aliaksandr Valialkin	539fce9227	lib/logstorage: work-in-progress	2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin	b30e80b071	lib/logstorage: work-in-progress	2024-05-30 16:19:23 +02:00
Aliaksandr Valialkin	1de187bcb7	lib/logstorage: work-in-progress	2024-05-29 01:52:13 +02:00
Aliaksandr Valialkin	0aafca29be	lib/logstorage: work-in-progress	2024-05-28 19:29:41 +02:00
Aliaksandr Valialkin	99138e15c0	lib/logstorage: fix golangci-lint warnings	2024-05-26 02:01:32 +02:00
Aliaksandr Valialkin	1e203f35f7	lib/logstorage: work-in-progress	2024-05-26 01:55:21 +02:00
Aliaksandr Valialkin	7ac529c235	lib/logstorage: work-in-progress	2024-05-25 22:59:13 +02:00
Aliaksandr Valialkin	0b629ce5a5	lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll	2024-05-25 22:13:32 +02:00
Aliaksandr Valialkin	dc55146752	lib/logstorage: work-in-progress	2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin	e2590f0485	lib/logstorage: work-in-progress	2024-05-25 00:30:58 +02:00
Aliaksandr Valialkin	4b458370c1	lib/logstorage: work-in-progress	2024-05-24 03:06:55 +02:00
Alexander Marshalov	7da541360e	[vmlogs] fixed time parsing with millisecond precision time (#6293) (#6295) ... fix for #6293 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-05-22 21:46:50 +02:00
Aliaksandr Valialkin	22107421eb	lib/logstorage: work-in-progress	2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin	bc4a0b8f37	lib/logstorage: fix golangci-lint warnings	2024-05-20 11:04:12 +02:00
Aliaksandr Valialkin	ad505a7a9a	lib/logstorage: work-in-progress	2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin	0aa19a2837	lib/logstorage: work-in-progress	2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin	da3af090c6	lib/logstorage: work-in-progress	2024-05-14 03:05:03 +02:00
Aliaksandr Valialkin	cb35e62e04	lib/logstorage: work-in-progress ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258	2024-05-14 01:49:23 +02:00
Aliaksandr Valialkin	cc2647d212	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit ... Change the return values for these functions - now they return the unmarshaled result plus the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling. This improves performance of these functions in hot loops of VictoriaLogs a bit.	2024-05-14 01:23:54 +02:00
hagen1778	17283fab6c	lib/logstorage: make linter happy ... Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-13 15:35:11 +02:00
Aliaksandr Valialkin	9dbd0f9085	lib/logstorage: initial implementation of pipes in LogsQL ... See https://docs.victoriametrics.com/victorialogs/logsql/#pipes	2024-05-12 16:33:31 +02:00
Aliaksandr Valialkin	590160ddbb	lib/slicesutil: add helper functions for setting slice length and extending its capacity ... The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.	2024-05-12 11:32:17 +02:00
wanshuangcheng	83216e956c	chore: fix function names in comment (#6076) ... Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>	2024-04-08 01:11:12 -07:00
Aliaksandr Valialkin	918cccaddf	all: fix golangci-lint(revive) warnings after 0c0ed61ce7 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001	2024-04-02 23:16:29 +03:00
XLONG96	a5795f533d	lib/logstorage: avoid panic when parsing regex with stream filter (#5897)	2024-02-29 15:31:54 +02:00
Aliaksandr Valialkin	4617dc8bbe	lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types ... See ea9e2b19a5	2024-02-23 23:46:13 +02:00
Aliaksandr Valialkin	f81b480905	lib/mergeset: consistently use atomic.* types instead of atomic.* function calls on ordinary types ... See ea9e2b19a5	2024-02-23 23:29:35 +02:00
Aliaksandr Valialkin	275335c181	lib/logstorage: consistently use atomic.* type for refCount and mustDrop fields in datadb and storage structs in the same way as it is used in lib/storage ... See ea9e2b19a5 and a204fd69f1	2024-02-23 23:04:42 +02:00
Aliaksandr Valialkin	0514091948	app/vlselect: follow-up for 451d2abf50 ... - Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb. See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request. - Document the change at docs/VictoriaLogs/CHANGELOG.md - Document the `limit` query arg at docs/VictoriaLogs/querying/README.md - Make the change less intrusive. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778	2024-02-18 23:05:51 +02:00
Dmytro Kozlov	451d2abf50	Enable the limit query param for the /select/logsql/query (#5778) ... * app/vlselect: add limit for logs query * app/vlselect: CHANGELOG.md * app/vlselect: stop search process if limit is reached, update logic, remove default limit * app/vlselect: fix tests * app/vlselect: fix filter tests * app/vlselect: fix tests	2024-02-18 22:58:47 +02:00
noodles2hg	cafd6f08b3	lib/logstorage: proper exit during block search (#5400)	2024-02-01 12:11:05 +00:00