From a9a26c20b5da45f687da93582c7b9a3b032c40a1 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Fri, 10 Nov 2023 20:58:32 +0100 Subject: [PATCH] docs/Single-server-VictoriaMetrics.md: make `High availability` section more clear --- README.md | 55 +++++++++++----------- docs/README.md | 67 ++++++++++++++++----------- docs/Single-server-VictoriaMetrics.md | 67 ++++++++++++++++----------- docs/vmagent.md | 48 +++++++++---------- 4 files changed, 132 insertions(+), 105 deletions(-) diff --git a/README.md b/README.md index e4e99ae5b..c6b591e71 100644 --- a/README.md +++ b/README.md @@ -1695,43 +1695,44 @@ See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs ## High availability -* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). -* Pass addresses of these instances to [vmagent](https://docs.victoriametrics.com/vmagent.html) via `-remoteWrite.url` command-line flag: +The general approach for achieving high availability is the following: + +- to run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones) +- to store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus +- to query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable. + +Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable. +The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying. +Both [vmagent](https://docs.victoriametrics.com/vmagent.html) and Prometheus buffer the collected data locally if they cannot send it +to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance +after it becomes available. + +If you use [vmagent](https://docs.victoriametrics.com/vmagent.html) for storing the data into VictoriaMetrics, +then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics +instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances. +For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics: ```console -/path/to/vmagent -remoteWrite.url=http://:8428/api/v1/write -remoteWrite.url=http://:8428/api/v1/write +/path/to/vmagent \ + -remoteWrite.url=http://:8428/api/v1/write \ + -remoteWrite.url=http://:8428/api/v1/write ``` -Alternatively these addresses may be passed to `remote_write` section in Prometheus config: +If you use Prometheus for collecting and writing the data to VictoriaMetrics, +then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section +in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances: ```yml remote_write: - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 - # ... - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 + - url: http://:8428/api/v1/write + - url: http://:8428/api/v1/write ``` -* Apply the updated config: +It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups, +since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus. -```console -kill -HUP `pidof prometheus` -``` - -It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups. - -* Now Prometheus should write data into all the configured `remote_write` urls in parallel. -* Set up [Promxy](https://github.com/jacksontj/promxy) in front of all the VictoriaMetrics replicas. -* Set up Prometheus datasource in Grafana that points to Promxy. - -If you have Prometheus HA pairs with replicas `r1` and `r2` in each pair, then configure each `r1` -to write data to `victoriametrics-addr-1`, while each `r2` should write data to `victoriametrics-addr-2`. - -Another option is to write data simultaneously from Prometheus HA pair to a pair of VictoriaMetrics instances -with the enabled de-duplication. See [this section](#deduplication) for details. +If you use identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) instances for collecting the same data +and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side. ## Deduplication diff --git a/docs/README.md b/docs/README.md index 108e9678c..9fb8b2d6d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1129,6 +1129,18 @@ For example, the following command builds the image on top of [scratch](https:// ROOT_IMAGE=scratch make package-victoria-metrics ``` +#### Building VictoriaMetrics with Podman + +VictoriaMetrics can be built with Podman in either rootful or rootless mode. + +When building via rootlful Podman, simply add `DOCKER=podman` to the relevant `make` commandline. To build +via rootless Podman, add `DOCKER=podman DOCKER_RUN="podman run --userns=keep-id"` to the `make` +commandline. + +For example: `make victoria-metrics-pure DOCKER=podman DOCKER_RUN="podman run --userns=keep-id"` + +Note that `production` builds are not supported via Podman becuase Podman does not support `buildx`. + ## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) @@ -1686,43 +1698,44 @@ See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs ## High availability -* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). -* Pass addresses of these instances to [vmagent](https://docs.victoriametrics.com/vmagent.html) via `-remoteWrite.url` command-line flag: +The general approach for achieving high availability is the following: + +- to run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones) +- to store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus +- to query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable. + +Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable. +The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying. +Both [vmagent](https://docs.victoriametrics.com/vmagent.html) and Prometheus buffer the collected data locally if they cannot send it +to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance +after it becomes available. + +If you use [vmagent](https://docs.victoriametrics.com/vmagent.html) for storing the data into VictoriaMetrics, +then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics +instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances. +For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics: ```console -/path/to/vmagent -remoteWrite.url=http://:8428/api/v1/write -remoteWrite.url=http://:8428/api/v1/write +/path/to/vmagent \ + -remoteWrite.url=http://:8428/api/v1/write \ + -remoteWrite.url=http://:8428/api/v1/write ``` -Alternatively these addresses may be passed to `remote_write` section in Prometheus config: +If you use Prometheus for collecting and writing the data to VictoriaMetrics, +then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section +in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances: ```yml remote_write: - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 - # ... - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 + - url: http://:8428/api/v1/write + - url: http://:8428/api/v1/write ``` -* Apply the updated config: +It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups, +since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus. -```console -kill -HUP `pidof prometheus` -``` - -It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups. - -* Now Prometheus should write data into all the configured `remote_write` urls in parallel. -* Set up [Promxy](https://github.com/jacksontj/promxy) in front of all the VictoriaMetrics replicas. -* Set up Prometheus datasource in Grafana that points to Promxy. - -If you have Prometheus HA pairs with replicas `r1` and `r2` in each pair, then configure each `r1` -to write data to `victoriametrics-addr-1`, while each `r2` should write data to `victoriametrics-addr-2`. - -Another option is to write data simultaneously from Prometheus HA pair to a pair of VictoriaMetrics instances -with the enabled de-duplication. See [this section](#deduplication) for details. +If you use identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) instances for collecting the same data +and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side. ## Deduplication diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index 6c8b4e6b6..a29541ebb 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -1137,6 +1137,18 @@ For example, the following command builds the image on top of [scratch](https:// ROOT_IMAGE=scratch make package-victoria-metrics ``` +#### Building VictoriaMetrics with Podman + +VictoriaMetrics can be built with Podman in either rootful or rootless mode. + +When building via rootlful Podman, simply add `DOCKER=podman` to the relevant `make` commandline. To build +via rootless Podman, add `DOCKER=podman DOCKER_RUN="podman run --userns=keep-id"` to the `make` +commandline. + +For example: `make victoria-metrics-pure DOCKER=podman DOCKER_RUN="podman run --userns=keep-id"` + +Note that `production` builds are not supported via Podman becuase Podman does not support `buildx`. + ## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) @@ -1694,43 +1706,44 @@ See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs ## High availability -* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). -* Pass addresses of these instances to [vmagent](https://docs.victoriametrics.com/vmagent.html) via `-remoteWrite.url` command-line flag: +The general approach for achieving high availability is the following: + +- to run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones) +- to store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus +- to query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable. + +Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable. +The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying. +Both [vmagent](https://docs.victoriametrics.com/vmagent.html) and Prometheus buffer the collected data locally if they cannot send it +to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance +after it becomes available. + +If you use [vmagent](https://docs.victoriametrics.com/vmagent.html) for storing the data into VictoriaMetrics, +then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics +instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances. +For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics: ```console -/path/to/vmagent -remoteWrite.url=http://:8428/api/v1/write -remoteWrite.url=http://:8428/api/v1/write +/path/to/vmagent \ + -remoteWrite.url=http://:8428/api/v1/write \ + -remoteWrite.url=http://:8428/api/v1/write ``` -Alternatively these addresses may be passed to `remote_write` section in Prometheus config: +If you use Prometheus for collecting and writing the data to VictoriaMetrics, +then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section +in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances: ```yml remote_write: - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 - # ... - - url: http://:8428/api/v1/write - queue_config: - max_samples_per_send: 10000 + - url: http://:8428/api/v1/write + - url: http://:8428/api/v1/write ``` -* Apply the updated config: +It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups, +since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus. -```console -kill -HUP `pidof prometheus` -``` - -It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent.html) instead of Prometheus for highly loaded setups. - -* Now Prometheus should write data into all the configured `remote_write` urls in parallel. -* Set up [Promxy](https://github.com/jacksontj/promxy) in front of all the VictoriaMetrics replicas. -* Set up Prometheus datasource in Grafana that points to Promxy. - -If you have Prometheus HA pairs with replicas `r1` and `r2` in each pair, then configure each `r1` -to write data to `victoriametrics-addr-1`, while each `r2` should write data to `victoriametrics-addr-2`. - -Another option is to write data simultaneously from Prometheus HA pair to a pair of VictoriaMetrics instances -with the enabled de-duplication. See [this section](#deduplication) for details. +If you use identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) instances for collecting the same data +and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side. ## Deduplication diff --git a/docs/vmagent.md b/docs/vmagent.md index 3a71b0695..d6ae0b1d7 100644 --- a/docs/vmagent.md +++ b/docs/vmagent.md @@ -71,7 +71,7 @@ and sending the data to the Prometheus-compatible remote storage: Example command for writing the data received via [supported push-based protocols](#how-to-push-data-to-vmagent) to [single-node VictoriaMetrics](https://docs.victoriametrics.com/) located at `victoria-metrics-host:8428`: -```bash +```console /path/to/vmagent -remoteWrite.url=https://victoria-metrics-host:8428/api/v1/write ``` @@ -80,7 +80,7 @@ the data to [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-V Example command for scraping Prometheus targets and writing the data to single-node VictoriaMetrics: -```bash +```console /path/to/vmagent -promscrape.config=/path/to/prometheus.yml -remoteWrite.url=https://victoria-metrics-host:8428/api/v1/write ``` @@ -121,7 +121,7 @@ additionally to pull-based Prometheus-compatible targets' scraping: * Sending `SIGHUP` signal to `vmagent` process: - ```bash + ```console kill -SIGHUP `pidof vmagent` ``` @@ -336,7 +336,7 @@ in the `scrape_config_files` section of `-promscrape.config` file. For example, loading scrape configs from all the `*.yml` files under `configs` directory, from `single_scrape_config.yml` local file and from `https://config-server/scrape_config.yml` url: -```yaml +```yml scrape_config_files: - configs/*.yml - single_scrape_config.yml @@ -346,7 +346,7 @@ scrape_config_files: Every referred file can contain arbitrary number of [supported scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). There is no need in specifying top-level `scrape_configs` section in these files. For example: -```yaml +```yml - job_name: foo static_configs: - targets: ["vmagent:8429"] @@ -386,7 +386,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me For example, the following command starts `vmagent`, which adds `{datacenter="foobar"}` label to all the metrics pushed to all the configured remote storage systems (all the `-remoteWrite.url` flag values): - ```bash + ``` /path/to/vmagent -remoteWrite.label=datacenter=foobar ... ``` @@ -751,7 +751,7 @@ stream parsing mode can be explicitly enabled in the following places: Examples: -```yaml +```yml scrape_configs: - job_name: 'big-federate' stream_parse: true @@ -778,7 +778,7 @@ Each `vmagent` instance in the cluster must use identical `-promscrape.config` f in the range `0 ... N-1`, where `N` is the number of `vmagent` instances in the cluster specified via `-promscrape.cluster.membersCount`. For example, the following commands spread scrape targets among a cluster of two `vmagent` instances: -```text +``` /path/to/vmagent -promscrape.cluster.membersCount=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... ``` @@ -790,7 +790,7 @@ By default, each scrape target is scraped only by a single `vmagent` instance in then `-promscrape.cluster.replicationFactor` command-line flag must be set to the desired number of replicas. For example, the following commands start a cluster of three `vmagent` instances, where each target is scraped by two `vmagent` instances: -```text +``` /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ... @@ -804,7 +804,7 @@ The `-promscrape.cluster.memberLabel` command-line flag allows specifying a name The value of the `member num` label is set to `-promscrape.cluster.memberNum`. For example, the following config instructs adding `vmagent_instance="0"` label to all the metrics scraped by the given `vmagent` instance: -```text +``` /path/to/vmagent -promscrape.cluster.membersCount=2 -promscrape.cluster.memberNum=0 -promscrape.cluster.memberLabel=vmagent_instance ``` @@ -831,7 +831,7 @@ See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679) `vmagent` supports scraping targets via http, https and socks5 proxies. Proxy address must be specified in `proxy_url` option. For example, the following scrape config instructs target scraping via https proxy at `https://proxy-addr:1234`: -```yaml +```yml scrape_configs: - job_name: foo proxy_url: https://proxy-addr:1234 @@ -848,7 +848,7 @@ Proxy can be configured with the following optional settings: For example: -```yaml +```yml scrape_configs: - job_name: foo proxy_url: https://proxy-addr:1234 @@ -998,7 +998,7 @@ If you have suggestions for improvements or have found a bug - please open an is * By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at the beginning of some interval, then `scrape_align_interval` option must be used. For example, the following config aligns hourly scrapes to the beginning of hour: - ```yaml + ```yml scrape_configs: - job_name: foo scrape_interval: 1h @@ -1008,7 +1008,7 @@ If you have suggestions for improvements or have found a bug - please open an is * By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at specific offset, then `scrape_offset` option must be used. For example, the following config instructs `vmagent` to scrape the target at 10 seconds of every minute: - ```yaml + ```yml scrape_configs: - job_name: foo scrape_interval: 1m @@ -1021,14 +1021,14 @@ If you have suggestions for improvements or have found a bug - please open an is The following relabeling rule may be added to `relabel_configs` section in order to filter out pods with unneeded ports: - ```yaml + ```yml - action: keep_if_equal source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number] ``` The following relabeling rule may be added to `relabel_configs` section in order to filter out init container pods: - ```yaml + ```yml - action: drop source_labels: [__meta_kubernetes_pod_container_init] regex: true @@ -1072,7 +1072,7 @@ For example, `-kafka.consumer.topic.brokers=host1:9092;host2:9092`. The following command starts `vmagent`, which reads metrics in InfluxDB line protocol format from Kafka broker at `localhost:9092` from the topic `metrics-by-telegraf` and sends them to remote storage at `http://localhost:8428/api/v1/write`: -```bash +```console ./bin/vmagent -remoteWrite.url=http://localhost:8428/api/v1/write \ -kafka.consumer.topic.brokers=localhost:9092 \ -kafka.consumer.topic.format=influx \ @@ -1095,7 +1095,7 @@ These command-line flags are available only in [enterprise](https://docs.victori which can be downloaded for evaluation from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) page (see `vmutils-...-enterprise.tar.gz` archives) and from [docker images](https://hub.docker.com/r/victoriametrics/vmagent/tags) with tags containing `enterprise` suffix. -```text +``` -kafka.consumer.topic array Kafka topic names for data consumption. Supports an array of values separated by comma or specified via multiple flags. @@ -1140,13 +1140,13 @@ Two types of auth are supported: * sasl with username and password: -```bash +```console ./bin/vmagent -remoteWrite.url=kafka://localhost:9092/?topic=prom-rw&security.protocol=SASL_SSL&sasl.mechanisms=PLAIN -remoteWrite.basicAuth.username=user -remoteWrite.basicAuth.password=password ``` * tls certificates: -```bash +```console ./bin/vmagent -remoteWrite.url=kafka://localhost:9092/?topic=prom-rw&security.protocol=SSL -remoteWrite.tlsCAFile=/opt/ca.pem -remoteWrite.tlsCertFile=/opt/cert.pem -remoteWrite.tlsKeyFile=/opt/key.pem ``` @@ -1177,7 +1177,7 @@ The `` may be manually set via `PKG_TAG=foobar make package-vmagent`. The base docker image is [alpine](https://hub.docker.com/_/alpine) but it is possible to use any other base image by setting it via `` environment variable. For example, the following command builds the image on top of [scratch](https://hub.docker.com/_/scratch) image: -```bash +```console ROOT_IMAGE=scratch make package-vmagent ``` @@ -1205,7 +1205,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b
-```bash +```console curl http://0.0.0.0:8429/debug/pprof/heap > mem.pprof ``` @@ -1215,7 +1215,7 @@ curl http://0.0.0.0:8429/debug/pprof/heap > mem.pprof
-```bash +```console curl http://0.0.0.0:8429/debug/pprof/profile > cpu.pprof ``` @@ -1231,7 +1231,7 @@ It is safe sharing the collected profiles from security point of view, since the `vmagent` can be fine-tuned with various command-line flags. Run `./vmagent -help` in order to see the full list of these flags with their descriptions and default values: -```text +``` ./vmagent -help vmagent collects metrics data via popular data ingestion protocols and routes them to VictoriaMetrics.