12 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	66b2987f49	lib/logstorage: optimize query imeediately after its parsing ... This eliminates possible bugs related to forgotten Query.Optimize() calls. This also allows removing optimize() function from pipe interface. While at it, drop filterNoop inside filterAnd.	2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin	7a623c225f	lib/logstorage: follow-up for af831a6c906158f371f1b6810706fa0a54b78386 ... Sync the code between top and sort pipes regarding the code related to rank.	2024-10-29 16:44:46 +01:00
Aliaksandr Valialkin	3c06d083ea	lib/logstorage: add an ability to return rank from top pipe results	2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin	c4b2fdff70	lib/logstorage: optimize 'stats by(...)' calculations for by(...) fields with millions of unique values on multi-CPU systems ... - Parallelize merging of per-CPU `stats by(...)` result shards. - Parallelize writing `stats by(...)` results to the next pipe.	2024-10-18 02:22:41 +02:00
Aliaksandr Valialkin	192c07f76a	lib/logstorage: optimize performance for top pipe when it is applied to a field with millions of unique values ... - Use parallel merge of per-CPU shard results. This improves merge performance on multi-CPU systems. - Use topN heap sort of per-shard results. This improves performance when results contain millions of entries.	2024-10-18 02:21:56 +02:00
Aliaksandr Valialkin	a350be48b6	lib/logstorage: do not count dictionary values which have no matching logs in count_uniq stats function ... Create blockResultColumn.forEachDictValue* helper functions for visiting matching dictionary values. These helper functions should prevent from counting dictionary values without matching logs in the future. This is a follow-up for 0c0f013a60 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7152	2024-10-01 13:34:45 +02:00
Aliaksandr Valialkin	0c0f013a60	lib/logstorage: skip values with zero hits for 'uniq', 'top' and 'field_values' pipes ... See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/72#issuecomment-2352078483	2024-09-30 14:15:07 +02:00
Aliaksandr Valialkin	55eb321f77	lib/logstorage: clear hits slice obtained from encoding.GetUint64s() before updating it with hits for valueTypeDict column ... encoding.GetUint64s() returns uninitialized slice, which may contain arbitrary values. So values in this slice must be reset to zero before using it for counting hits in `uniq` and `top` pipes.	2024-09-29 10:29:13 +02:00
Aliaksandr Valialkin	94afcbd9a9	lib/logstorage: postpone initialization of per-shard stateSizeBudget until the first call to pipeProcessor.writeBlock() ... This simplifies pipeProcessor initialization logic a bit. This also doesn't mangle the original maxStateSize value, which is used in error messages when the state size exceeds maxStateSize.	2024-09-29 10:29:13 +02:00
Aliaksandr Valialkin	4599429f51	lib/logstorage: read timestamps column when it is really needed during query execution ... Previously timestamps column was read unconditionally on every query. This could significantly slow down queries, which do not need reading this column like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070 .	2024-09-25 19:17:47 +02:00
Aliaksandr Valialkin	87f1c8bd6c	lib/logstorage: work-in-progress	2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin	2b6a634ec0	lib/logstorage: work-in-progress	2024-06-17 12:13:18 +02:00