mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-18 22:52:11 +01:00
c6e0780f4b
This change is made in attempt to reduce memory usage by vmalert when parsing big instant responses from VM/Prometheus. Ina5c427bac4vmalert switched from std json lib to fastjson lib in order to reduce amount of allocations, as according to highloaded profiles of vmalert the CPU is mostly spent on GC. But switching to fastjson resulted into excessive memory usage for cases when vmalert has to parse long json lines, which usually happens when instant response contains many `metric` objects. In this change we do a mixed parsing: 1. Slice of `metric` objects is parsed with std lib to keep mem low 2. Each `metric` object is parsed with fastjson to reduce allocs The benchmark results are the following: ``` pkg: github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource BenchmarkParsePrometheusResponse/Instant_std+fastjson-10 1760 668959 ns/op 280147 B/op 5781 allocs/op MBs allocated at heap: 493.078392 mallocs: 18655472 BenchmarkParsePrometheusResponse/Instant_fastjson-10 6109 198258 ns/op 172839 B/op 5548 allocs/op MBs allocated at heap: 1056.384464 mallocs: 34457184 BenchmarkParsePrometheusResponse/Instant_std-10 1287 950987 ns/op 451677 B/op 9619 allocs/op MBs allocated at heap: 580.802976 mallocs: 13351636 ``` The benchmark function code with mem measurement is available here https://gist.github.com/hagen1778/b9c3ca7f8ca7d6b21aec9777112c5810 The benchmark contains 3 results: 1. Instant_std+fastjson is the implementation in this change 2. Instant_fastjson-10 is the implementation froma5c427bac43. BenchmarkParsePrometheusResponse/Instant_std-10 is implementation beforea5c427bac4According to these results, this new implementation is slower than previous, but faster than before switching to fastjson. It also has lower number of allocations and roughly the same memory allocation on heap with GC turned off. --------- Other changes: 1. rm BenchmarkMetrics as it doesn't measure anything 2. simplify BenchmarkParsePrometheusResponse into BenchmarkPromInstantUnmarshal ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are **mandatory**: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: hagen1778 <roman@victoriametrics.com>
25 lines
591 B
Go
25 lines
591 B
Go
package datasource
|
|
|
|
import (
|
|
"os"
|
|
"testing"
|
|
)
|
|
|
|
func BenchmarkPromInstantUnmarshal(b *testing.B) {
|
|
data, err := os.ReadFile("testdata/instant_response.json")
|
|
if err != nil {
|
|
b.Fatalf("error while reading file: %s", err)
|
|
}
|
|
|
|
// BenchmarkParsePrometheusResponse/Instant_std+fastjson-10 1760 668959 ns/op 280147 B/op 5781 allocs/op
|
|
b.Run("Instant std+fastjson", func(b *testing.B) {
|
|
for i := 0; i < b.N; i++ {
|
|
var pi promInstant
|
|
err = pi.Unmarshal(data)
|
|
if err != nil {
|
|
b.Fatalf("unexpected parse err: %s", err)
|
|
}
|
|
}
|
|
})
|
|
}
|