VictoriaMetrics/lib/streamaggr
Roman Khavronenko 78121642df
lib/streamaggr: reduce number of inuse objects (#6402)
The main change is getting rid of interning of sample key. It was
discovered that for cases with many unique time series aggregated by
vmagent interned keys could grow up to hundreds of millions of objects.
This has negative impact on the following aspects:
1. It slows down garbage collection cycles, as GC has to scan all inuse
objects periodically. The higher is the number of inuse objects, the
longer it takes/the more CPU it takes.
2. It slows down the hot path of samples aggregation where each key
needs to be looked up in the map first.

The change makes code more fragile, but suppose to provide performance
optimization for heavy-loaded vmagents with stream aggregation enabled.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-06-07 16:35:52 +02:00
..
avg.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
count_samples.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
count_series.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
dedup_test.go lib/streamaggr: pick sample with bigger timestamp or value on deduplicator (#5939) 2024-03-17 23:06:37 +02:00
dedup_timing_test.go streamaggr: made labels compressor shared (#6173) 2024-05-10 14:28:59 +02:00
dedup.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
deduplicator_test.go lib/streamaggr: follow-up for 15e33d56f1 2024-03-17 23:23:57 +02:00
deduplicator_timing_test.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-03 03:00:45 +03:00
deduplicator.go streamaggr: made labels compressor shared (#6173) 2024-05-10 14:28:59 +02:00
histogram_bucket.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
last.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
max.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
min.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
quantiles.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
rate.go lib/streamaggr: added rate_sum and rate_avg to benchmarks, lint fix (#6264) 2024-05-13 16:49:59 +02:00
stddev.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
stdvar.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
streamaggr_test.go lib/streamaggr: added rate and rate_avg output (#6243) 2024-05-13 16:49:39 +02:00
streamaggr_timing_test.go lib/streamaggr: added rate_sum and rate_avg to benchmarks, lint fix (#6264) 2024-05-13 16:49:59 +02:00
streamaggr.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
sum_samples.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
total.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00
unique_samples.go lib/streamaggr: reduce number of inuse objects (#6402) 2024-06-07 16:35:52 +02:00