VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin f5dfe1cacd
lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe
Previously only logs inside the selected time range could be returned by stream_context pipe.
For example, the following query could return up to 10 surrounding logs only for the last 5 minutes,
while most users expect this query should return up to 10 surrounding logs without restrictions on the time range.

    _time:5m panic | stream_context before 10

This enables the ability to implement stream context feature at VictoriaLogs web UI: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7063 .

Reduce memory usage when returning stream context over big log streams with millions of entries.
The new logic scans over all the log messages for the selected log stream, while keeping in memory only
the given number of surrounding logs. Previously all the logs for the given log stream on the selected time range
were loaded in memory before selecting the needed surrounding logs.
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6730 .

Reduce the scan performance for big log streams by fetching only the requested fields. For example, the following
query should be executed much faster than before if logs contain many fields other than _stream, _msg and _time:

    panic | stream_context after 30 | fields _stream, _msg, _time
2024-09-26 17:04:39 +02:00
..
arena_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
arena.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
bitmap_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
bitmap_timing_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
bitmap.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
block_data_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_data.go lib/logstorage: work-in-progress 2024-05-14 01:49:58 +02:00
block_header_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
block_result.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
block_search.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
block_stream_reader.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_stream_writer.go chore: fix function names in comment (#6076) 2024-04-08 15:38:51 +02:00
block_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_timing_test.go
block.go lib/logstorage: work-in-progress 2024-05-14 01:49:58 +02:00
bloomfilter_test.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:42 +02:00
bloomfilter.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:42 +02:00
consts.go lib/logstorage/consts.go: document that it isn't recommended setting maxColumnsPerBlock constant to too big values 2024-09-24 18:52:54 +02:00
datadb_test.go
datadb.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-06 22:22:08 -07:00
encoding_timing_test.go
encoding.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
fields_set_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
fields_set.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
filenames.go
filter_and_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:28:34 +02:00
filter_and.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_any_case_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_any_case_phrase.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_any_case_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_any_case_prefix.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_day_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_day_range.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
filter_exact_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_exact_prefix.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_exact_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_exact.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_in_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_in.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_ipv4_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_ipv4_range.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_len_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_len_range.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_noop.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
filter_not_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_not.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_or_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:28:34 +02:00
filter_or.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_phrase.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_prefix.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_range.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_regexp_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_regexp.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_sequence_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_sequence.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_stream_id_test.go lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
filter_stream_id.go lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
filter_stream.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
filter_string_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_string_range.go lib/logstorage: lazily read column headers metadata during queries 2024-09-25 19:18:37 +02:00
filter_test.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
filter_time_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_time.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
filter_timing_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
filter_week_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_week_range.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
filter.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
hash128_test.go
hash128_timing_test.go lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types 2024-02-24 00:29:39 +02:00
hash128.go
if_filter.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
index_block_header_test.go
index_block_header.go
indexdb_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
indexdb.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-06 22:24:18 -07:00
inmemory_part_timing_test.go
inmemory_part.go
json_parser_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
json_parser.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
log_rows_timing_test.go
log_rows.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
logfmt_parser_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
logfmt_parser.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
parser_test.go app/vlinsert: support _time field without timezone information during data ingestion 2024-09-26 12:50:14 +02:00
parser.go app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop 2024-09-26 08:57:48 +02:00
part_header_test.go
part_header.go
part.go
partition_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
partition.go lib/logstorage: improve the performance of obtaining _stream column value 2024-09-24 20:57:39 +02:00
pattern_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pattern_timing_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pattern.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pipe_blocks_count_test.go lib/logstorage: add blocks_count pipe 2024-09-25 19:18:38 +02:00
pipe_blocks_count.go lib/logstorage: add blocks_count pipe 2024-09-25 19:18:38 +02:00
pipe_copy_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_copy.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_delete_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_delete.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_drop_empty_fields_test.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
pipe_drop_empty_fields.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_extract_regexp_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_extract_regexp.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_extract_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_extract.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_field_names_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pipe_field_names.go lib/logstorage: add blocks_count pipe 2024-09-25 19:18:38 +02:00
pipe_field_values_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_field_values.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_fields_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_fields.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_filter_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
pipe_filter.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_format_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
pipe_format.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_limit_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_limit.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_math_test.go lib/logstorage: substitute | operator with or operator at math pipe 2024-09-06 22:43:29 +02:00
pipe_math.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_offset_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_offset.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_pack_json_test.go lib/logstorage: work-in-progress 2024-05-29 01:52:34 +02:00
pipe_pack_json.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_pack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
pipe_pack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_pack.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_rename_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_rename.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_replace_regexp_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_replace_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_replace_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_replace.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_sort_test.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:46:03 +02:00
pipe_sort_topk.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_sort.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_stats_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
pipe_stats.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_stream_context_test.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:39:45 +02:00
pipe_stream_context.go lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe 2024-09-26 17:04:39 +02:00
pipe_top_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
pipe_top.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_uniq_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_uniq.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_unpack_json_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_unpack_json.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
pipe_unpack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack_syslog_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
pipe_unpack_syslog.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_unroll_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
pipe_unroll.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_update.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe_utils_test.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
pipe.go lib/logstorage: add blocks_count pipe 2024-09-25 19:18:38 +02:00
rows_test.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
rows.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
stats_avg_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_avg.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_empty_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_empty.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_count_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_uniq_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_uniq.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_count.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_max_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_max.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_median_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_median.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_min_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_min.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_quantile_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_quantile.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_row_any_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_row_any.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_row_max_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_row_max.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_row_min_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_row_min.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
stats_sum_len_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum_len.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_uniq_values_test.go lib/logstorage: work-in-progress 2024-05-25 00:31:55 +02:00
stats_uniq_values.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
stats_values_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_values.go lib/logstorage: read timestamps column when it is really needed during query execution 2024-09-25 19:18:37 +02:00
storage_search_test.go lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe 2024-09-26 17:04:39 +02:00
storage_search.go lib/logstorage: properly return surrounding logs outside the selected time range by stream_context pipe 2024-09-26 17:04:39 +02:00
storage_test.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:18:15 +02:00
storage.go lib/logstorage: improve the performance of obtaining _stream column value 2024-09-24 20:57:39 +02:00
stream_filter_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
stream_filter.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
stream_id_test.go lib/logstorage: add benchmark for streamID.marshalString 2024-09-24 18:38:21 +02:00
stream_id_timing_test.go lib/logstorage: add benchmark for streamID.marshalString 2024-09-24 18:38:21 +02:00
stream_id.go lib/logstorage: improve performance for streamID.marshalString() by more than 2x 2024-09-24 18:38:21 +02:00
stream_tags.go app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:55 +02:00
stringbucket.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
syslog_parser_test.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
syslog_parser.go app/vlinsert: support _time field without timezone information during data ingestion 2024-09-26 12:50:14 +02:00
tenant_id_test.go lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:39:13 +02:00
tenant_id.go lib/logstorage: improve performance for streamID.marshalString() by more than 2x 2024-09-24 18:38:21 +02:00
tokenizer_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
tokenizer_timing_test.go
tokenizer.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
u128_test.go
u128.go lib/logstorage: improve performance for streamID.marshalString() by more than 2x 2024-09-24 18:38:21 +02:00
values_encoder_test.go app/vlinsert: support _time field without timezone information during data ingestion 2024-09-26 12:50:14 +02:00
values_encoder_timing_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
values_encoder.go app/vlinsert: support _time field without timezone information during data ingestion 2024-09-26 12:50:14 +02:00