VictoriaMetrics: fast, cost-effective monitoring solution and time series database
Go to file
Roman Khavronenko c6e0780f4b
app/vmalert: update parsing for instant responses (#6859)
This change is made in attempt to reduce memory usage by vmalert when
parsing big instant responses from VM/Prometheus.

In
a5c427bac4
vmalert switched from std json lib to fastjson lib in order to reduce
amount of allocations, as according to highloaded profiles of vmalert
the CPU is mostly spent on GC.

But switching to fastjson resulted into excessive memory usage for cases
when vmalert has to parse long json lines, which usually happens when
instant response contains many `metric` objects.

In this change we do a mixed parsing:
1. Slice of `metric` objects is parsed with std lib to keep mem low
2. Each `metric` object is parsed with fastjson to reduce allocs

The benchmark results are the following:
```
pkg: github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource
BenchmarkParsePrometheusResponse/Instant_std+fastjson-10                    1760            668959 ns/op          280147 B/op       5781 allocs/op
MBs allocated at heap: 493.078392
mallocs: 18655472
BenchmarkParsePrometheusResponse/Instant_fastjson-10                        6109            198258 ns/op          172839 B/op       5548 allocs/op
MBs allocated at heap: 1056.384464
mallocs: 34457184
BenchmarkParsePrometheusResponse/Instant_std-10                             1287            950987 ns/op          451677 B/op       9619 allocs/op
MBs allocated at heap: 580.802976
mallocs: 13351636
```
The benchmark function code with mem measurement is available here
https://gist.github.com/hagen1778/b9c3ca7f8ca7d6b21aec9777112c5810

The benchmark contains 3 results:
1. Instant_std+fastjson is the implementation in this change
2. Instant_fastjson-10 is the implementation from
a5c427bac4
3. BenchmarkParsePrometheusResponse/Instant_std-10 is implementation
before
a5c427bac4

According to these results, this new implementation is slower than
previous, but faster than before switching to fastjson. It also has
lower number of allocations and roughly the same memory allocation on
heap with GC turned off.

---------

Other changes:
1. rm BenchmarkMetrics as it doesn't measure anything
2. simplify BenchmarkParsePrometheusResponse into
BenchmarkPromInstantUnmarshal

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 23:56:11 +02:00
.github Revert "deployment: build image for vmagent streamaggr benchmark (#6515)" 2024-07-16 13:34:16 +02:00
app app/vmalert: update parsing for instant responses (#6859) 2024-08-22 23:56:11 +02:00
cspell docs: add spellcheck command (#6562) 2024-07-11 12:40:24 +02:00
dashboards dashboards: add Scrape duration 0.99 quantile panel 2024-07-31 16:09:13 +02:00
deployment deployment: update compose images to v1.102.1 release 2024-08-02 11:29:48 +02:00
docs app/vmalert: update parsing for instant responses (#6859) 2024-08-22 23:56:11 +02:00
lib lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
package/release simplify release process (#3012) 2022-08-31 02:28:45 +03:00
vendor lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
.dockerignore added packer build for DigitalOcean Droplets (#1917) 2021-12-21 12:10:08 +02:00
.gitignore docs: removed redundant 'VictoriaLogs' from title (#6715) 2024-07-29 14:30:38 +02:00
.golangci.yml .golangci.yml: properly specify functions to exclude for return values check after the upgrade to v1.59.1 at 239a7b6e6f 2024-06-11 17:05:17 +02:00
.wwhrd.yml .wwhrd.yml: add ISC license, which is used by github.com/davecgh/go-spew 2022-12-14 14:56:06 -08:00
CODE_OF_CONDUCT.md A good change for MD files (#2353) 2022-03-22 14:01:04 +02:00
CONTRIBUTING.md Move CONTRIBUTING.md to docs/ 2024-04-20 23:12:53 +02:00
go.mod lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
go.sum lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
LICENSE LICENSE: update the current year from 2023 to 2024 2024-01-17 01:48:12 +02:00
logo.png Add logo 2018-11-29 21:47:17 +02:00
Makefile make: reset permissions when creating release archive (#6846) 2024-08-21 14:14:33 +02:00
README.md docs: rm mentions of google group as it was deprecated 2024-08-21 14:14:40 +02:00
SECURITY.md add new LTS release v1.102.x 2024-08-02 11:15:25 +02:00
VM_logo.zip removed DS_Store files from VM_logo.zip (#1233) 2021-04-20 11:46:57 +03:00

VictoriaMetrics

Latest Release Docker Pulls Slack GitHub license Go Report Build Status codecov

VictoriaMetrics logo

VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.

Here are some resources and information about VictoriaMetrics:

Yes, we open-source both the single-node VictoriaMetrics and the cluster version.

Prominent features

VictoriaMetrics is optimized for timeseries data, even when old time series are constantly replaced by new ones at a high rate, it offers a lot of features:

Enterprise version

In addition, the Enterprise version includes extra features:

  • Anomaly detection: Automation and simplification of your alerting rules, covering complex anomalies found in metrics data.
  • Backup automation: Automates regular backup procedures.
  • Multiple retentions: Reducing storage costs by specifying different retentions for different datasets.
  • Downsampling: Reducing storage costs and increasing performance for queries over historical data.
  • Stable releases with long-term support lines (LTS).
  • Comprehensive support: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
  • Many other features, which you can read about on the Enterprise page.

Contact us if you need enterprise support for VictoriaMetrics. Or you can request a free trial license here, downloaded Enterprise binaries are available at Github Releases.

We strictly apply security measures in everything we do. VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. See Security page for more details.

Benchmarks

Some good benchmarks VictoriaMetrics achieved:

Community and contributions

Feel free asking any questions regarding VictoriaMetrics:

If you like VictoriaMetrics and want to contribute, then please read these docs.

Zip contains three folders with different image orientations (main color and inverted version).

Files included in each folder:

  • 2 JPEG Preview files
  • 2 PNG Preview files with transparent background
  • 2 EPS Adobe Illustrator EPS10 files

Logo Usage Guidelines

Font used

  • Lato Black
  • Lato Regular

Color Palette

We kindly ask

  • Please don't use any other font instead of suggested.
  • To keep enough clear space around the logo.
  • Do not change spacing, alignment, or relative locations of the design elements.
  • Do not change the proportions for any of the design elements or the design itself. You may resize as needed but must retain all proportions.