From 91cbb9063d3384aafad8236997aab331081e01ef Mon Sep 17 00:00:00 2001 From: Nikolay Date: Wed, 15 Mar 2023 20:33:10 +0100 Subject: [PATCH] Vmagent kafka updates (#535) * app/vmagent: allow vm proto for kafka consumer and producer it should reduce network usage up to 50%. According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%. With enabled zstd at kafka topic, it shows no diffence in traffic. So it doesn't make much sense to use it. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225 * mention eb61a7dd68b834b08d01727a918f207700348ada at changelog * app/vmagent: bumps kafka lib version it allows compiling vmagent for arm64 machines fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271 * mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log * app/vmagent: adds natural concurrency for kafka consumer it should improve performance for data consumption https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957 * mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca * Update app/vmagent/kafka/consumer.go Co-authored-by: Roman Khavronenko * Update app/vmagent/kafka/consumer_cgo.go Co-authored-by: Roman Khavronenko --------- Co-authored-by: Aliaksandr Valialkin Co-authored-by: Roman Khavronenko --- deployment/docker/Makefile | 5 ----- docs/CHANGELOG.md | 3 +++ 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/deployment/docker/Makefile b/deployment/docker/Makefile index ab6d30fe5..8a16f2cda 100644 --- a/deployment/docker/Makefile +++ b/deployment/docker/Makefile @@ -162,12 +162,7 @@ package-via-docker-arm: GOARCH=arm $(MAKE) package-via-docker-goarch-nocgo package-via-docker-arm64: -ifeq ($(APP_NAME),vmagent) - GOARCH=arm64 $(MAKE) package-via-docker-goarch-nocgo -else $(MAKE) package-via-docker-goarch-arm64 -endif - package-via-docker-ppc64le: GOARCH=ppc64le $(MAKE) package-via-docker-goarch-nocgo diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index faa268393..2a0814656 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -16,6 +16,9 @@ The following tip changes can be tested by building VictoriaMetrics components f ## tip * FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds `-maxLabelValueLen`. This should simplify debugging for this case. +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent.html#victoriametrics-remote-write-protocol) when [sending / receiving data to / from Kafka](https://docs.victoriametrics.com/vmagent.html#kafka-integration). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `--kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [Kafka producer and consumer](https://docs.victoriametrics.com/vmagent.html#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271). ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1)