Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress
2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage
2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data
2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
...
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.
While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
Aliaksandr Valialkin
2a7fcba330
lib/logstorage: make golangci-lint happy
2024-06-25 03:06:28 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress
2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress
2024-06-20 03:10:37 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports
2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress
2024-06-17 12:13:25 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress
2024-06-11 17:51:01 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress
2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress
2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
a200fb433a
lib/logstorage: allow using eval keyword instead of math keyword in math pipe
2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress
2024-06-05 03:18:25 +02:00
Aliaksandr Valialkin
1ce8a9a751
lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc
2024-06-04 02:29:18 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress
2024-06-04 01:50:55 +02:00
Aliaksandr Valialkin
540bbb63a2
lib/logstorage: work-in-progress
2024-05-30 16:19:36 +02:00
Aliaksandr Valialkin
e83fd4a117
lib/logstorage: work-in-progress
2024-05-29 01:52:34 +02:00
Aliaksandr Valialkin
79c03fc35f
lib/logstorage: work-in-progress
2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin
ce5e4c842a
lib/logstorage: fix golangci-lint warnings
2024-05-26 02:02:41 +02:00
Aliaksandr Valialkin
afa597ce2a
lib/logstorage: work-in-progress
2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin
6427b3c3c0
lib/logstorage: work-in-progress
2024-05-25 22:59:21 +02:00
Aliaksandr Valialkin
9edbeca46b
lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll
2024-05-25 22:13:44 +02:00
Aliaksandr Valialkin
03fe4c8963
lib/logstorage: work-in-progress
2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress
2024-05-25 00:31:55 +02:00
Aliaksandr Valialkin
7a2a2f173e
lib/logstorage: work-in-progress
2024-05-24 03:07:07 +02:00
Alexander Marshalov
0b70c4c1f1
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:54:50 +02:00
Aliaksandr Valialkin
04d0dd2542
lib/logstorage: work-in-progress
2024-05-22 21:01:28 +02:00
Aliaksandr Valialkin
45fbcc74e0
lib/logstorage: fix golangci-lint warnings
2024-05-20 11:04:37 +02:00
Aliaksandr Valialkin
582e7d5439
lib/logstorage: work-in-progress
2024-05-20 04:09:15 +02:00
Aliaksandr Valialkin
28626db066
lib/logstorage: work-in-progress
...
(cherry picked from commit 0aa19a2837
2024-05-16 09:35:55 +02:00
Aliaksandr Valialkin
b1ee7bca1a
lib/logstorage: work-in-progress
2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin
f52275bbd7
lib/logstorage: work-in-progress
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258
2024-05-14 01:49:58 +02:00
Aliaksandr Valialkin
32193b6059
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit
...
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.
This improves performance of these functions in hot loops of VictoriaLogs a bit.
2024-05-14 01:30:25 +02:00
hagen1778
84a896cd6e
lib/logstorage: make linter happy
...
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 17283fab6c
2024-05-13 16:49:37 +02:00
Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin
87338633b1
lib/slicesutil: add helper functions for setting slice length and extending its capacity
...
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
2024-05-12 11:33:49 +02:00
wanshuangcheng
52a4ae0b28
chore: fix function names in comment ( #6076 )
...
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 15:38:51 +02:00
Aliaksandr Valialkin
00f59d6ddf
all: fix golangci-lint(revive) warnings after 0c0ed61ce7
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001
2024-04-03 03:00:45 +03:00
XLONG96
88b9088499
lib/logstorage: avoid panic when parsing regex with stream filter ( #5897 )
2024-02-29 15:32:25 +02:00
Aliaksandr Valialkin
ca1e78bd16
lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types
...
See ea9e2b19a5
2024-02-24 00:29:39 +02:00
Aliaksandr Valialkin
d0538d11d3
lib/mergeset: consistently use atomic.* types instead of atomic.* function calls on ordinary types
...
See ea9e2b19a5
2024-02-24 00:29:12 +02:00
Aliaksandr Valialkin
92e098012a
lib/logstorage: consistently use atomic.* type for refCount and mustDrop fields in datadb and storage structs in the same way as it is used in lib/storage
...
See ea9e2b19a5a204fd69f1
2024-02-24 00:28:56 +02:00
Aliaksandr Valialkin
b58c429044
app/vlselect: follow-up for 451d2abf50
...
- Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb.
See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request.
- Document the change at docs/VictoriaLogs/CHANGELOG.md
- Document the `limit` query arg at docs/VictoriaLogs/querying/README.md
- Make the change less intrusive.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778
2024-02-18 23:06:08 +02:00
Dmytro Kozlov
2d674f98d4
Enable the limit query param for the /select/logsql/query ( #5778 )
...
* app/vlselect: add limit for logs query
* app/vlselect: CHANGELOG.md
* app/vlselect: stop search process if limit is reached, update logic, remove default limit
* app/vlselect: fix tests
* app/vlselect: fix filter tests
* app/vlselect: fix tests
2024-02-18 22:59:16 +02:00
noodles2hg
60a8e59366
lib/logstorage: proper exit during block search ( #5400 )
2024-02-01 14:11:20 +02:00
Jiajing LU
9c75e3ee15
count inmemoryParts that have not been taken for merge ( #5447 )
2024-02-01 14:07:13 +02:00
Aliaksandr Valialkin
230ef43a32
lib/logstorage: make sure that WaitGroup.Add isnt called after stopCh is closed and WaitGroup.Wait is called
...
This protects from rare panic, which may occur during graceful shutdown of VictoriaLogs
2024-01-26 21:18:07 +01:00
Aliaksandr Valialkin
d52fd73f18
all: add up to 10% random jitter to the interval between periodic tasks performed by various components
...
This should smooth CPU and RAM usage spikes related to these periodic tasks,
by reducing the probability that multiple concurrent periodic tasks are performed at the same time.
2024-01-22 18:39:16 +02:00
Aliaksandr Valialkin
9760221214
lib/logstorage: always check the previous indexBlockHeader for blocks with matching tenantID and/or streamID
...
The previous indexBlockHeader may contain blocks for the matching tenantID and/or streamID,
so it must be scanned unconditionally during the search.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856
This is a follow-up for 89dcbc2fe7
2023-11-14 01:02:02 +01:00
