2471340e0d
Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333 |
||
---|---|---|
.. | ||
common | ||
deployment | ||
graphite | ||
influx | ||
opentsdb | ||
opentsdbhttp | ||
promremotewrite | ||
remotewrite | ||
vmimport | ||
main.go | ||
Makefile | ||
README.md | ||
vmagent.png |
vmagent
vmagent
is a tiny but brave agent, which helps you collecting metrics from various sources
and storing them to VictoriaMetrics.
Motivation
While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast
and RAM friendly to scrape metrics from Prometheus-compatible exporters to VictoriaMetrics.
Also, we found that users’ infrastructure is like snowflakes - never alike, and we decided to add more flexibility
to vmagent
(like the ability to push metrics instead of pulling them). We did our best and plan to do even more.
Features
- Can be used as drop-in replacement for Prometheus for scraping targets such as node_exporter. See Quick Start for details.
- Can add, remove and modify labels via Prometheus relabeling. See these docs for details.
- Accepts data via all the ingestion protocols supported by VictoriaMetrics:
- Influx line protocol via
http://<vmagent>:8429/write
. See these docs. - JSON lines import protocol via
http://<vmagent>:8429/api/v1/import
. See these docs. - Graphite plaintext protocol if
-graphiteListenAddr
command-line flag is set. See these docs. - OpenTSDB telnet and http protocols if
-opentsdbListenAddr
command-line flag is set. See these docs. - Prometheus remote write protocol via
http://<vmagent>:8429/api/v1/write
.
- Influx line protocol via
- Can replicate collected metrics simultaneously to multiple remote storage systems.
- Works in environments with unstable connections to remote storage. If the remote storage is unavailable, the collected metrics
are buffered at
-remoteWrite.tmpDataPath
. The buffered metrics are sent to remote storage as soon as connection to remote storage is recovered. - Uses lower amounts of RAM, CPU, disk IO and network bandwidth comparing to Prometheus.
Quick Start
Just download vmutils-*
archive from releases page, unpack it
and pass the following flags to vmagent
binary in order to start scraping Prometheus targets:
-promscrape.config
with the path to Prometheus config file (it is usually located at/etc/prometheus/prometheus.yml
)-remoteWrite.url
with the remote storage endpoint such as VictoriaMetrics. Multiple-remoteWrite.url
args can be set in parallel in order to replicate data concurrently to multiple remote storage systems.
Example command line:
/path/to/vmagent -promscrape.config=/path/to/prometheus.yml -remoteWrite.url=https://victoria-metrics-host:8428/api/v1/write
If you need collecting only Influx data, then the following command line would be enough:
/path/to/vmagent -remoteWrite.url=https://victoria-metrics-host:8428/api/v1/write
Then send Influx data to http://vmagent-host:8429/write
. See these docs for more details.
vmagent
is also available in docker images.
Pass -help
to vmagent
in order to see the full list of supported command-line flags with their descriptions.
How to collect metrics in Prometheus format?
Pass the path to prometheus.yml
to -promscrape.config
command-line flag. vmagent
takes into account the following
sections from Prometheus config file:
global
scrape_configs
All the other section are ignored, including remote_write section.
Use -remoteWrite.*
command-line flags instead for configuring remote write settings:
-remoteWrite.url
for pointing to remote storage. Data to remote storage can be sent either via HTTP or HTTPS. See-remoteWrite.tls*
flags for details.-remoteWrite.label
for adding labels to metrics before sending them to remote storage.-remoteWrite.relabelConfig
for applying relabeling to metrics before sending them to remote storage.
The following scrape types in scrape_config section are supported:
static_configs
- for scraping statically defined targets. See these docs for details.file_sd_configs
- for scraping targets defined in external files aka file-based service discover. See these docs for details.
File feature requests at our issue tracker if you need other service discovery mechanisms to be supported by vmagent
.
Adding labels to metrics
Labels can be added to metrics via the following mechanisms:
- Via
global -> external_labels
section in-promscrape.config
file. These labels are added only to metrics scraped from targets configured in-promscrape.config
file. - Via
-remoteWrite.label
command-line flag. These labels are added to all the collected metrics before sending them to-remoteWrite.url
.
Relabeling
vmagent
supports Prometheus relabeling.
Additionally it provides the following extra actions:
replace_all
: replaces all the occurences ofregex
in the values ofsource_labels
with thereplacement
and stores the result in thetarget_label
labelmap_all
: replaces all the occurences ofregex
in all the label names with thereplacement
The relabeling can be defined in the following places:
- At
scrape_config -> relabel_configs
section in-promscrape.config
file. This relabeling is applied to targets when parsing the file duringvmagent
startup or during config reload after sendingSIGHUP
signal tovmagent
viakill -HUP
. - At
scrape_config -> metric_relabel_configs
section in-promscrape.config
file. This relabeling is applied to metrics after each scrape for the configured targets. - At
-remoteWrite.relabelConfig
file. This relabeling is aplied to all the collected metrics before sending them to-remoteWrite.url
.
Read more about relabeling in the following articles:
- Life of a label
- Discarding targets and timeseries with relabeling
- Dropping labels at scrape time
- Extracting labels from legacy metric names
- relabel_configs vs metric_relabel_configs
Monitoring
vmagent
exports various metrics in Prometheus exposition format at http://vmagent-host:8429/metrics
page. It is recommended setting up regular scraping of this page
either via vmagent
itself or via Prometheus, so the exported metrics could be analyzed later.
vmagent
also exports target statuses at http://vmagent-host:8429/targets
page in plaintext format.
Troubleshooting
-
It is recommended increasing the maximum number of open files in the system (
ulimit -n
) when scraping big number of targets, sincevmagent
establishes at least a single TCP connection per each target. -
It is recommended increasing
-remoteWrite.queues
ifvmagent
collects more than 100K samples per second andvmagent_remotewrite_pending_data_bytes
metric exported byvmagent
at/metrics
page constantly grows. -
vmagent
buffers scraped data at-remoteWrite.tmpDataPath
directory until it is sent to-remoteWrite.url
. The directory can grow big when remote storage is unvailable during extended periods of time. If you don't want sending all the data from the directory to remote storage, just stopvmagent
and delete the directory.
How to build from sources
It is recommended using binary releases - vmagent
is located in vmutils-*
archives there.
Development build
- Install Go. The minimum supported version is Go 1.12.
- Run
make vmagent
from the root folder of the repository. It buildsvmagent
binary and puts it into thebin
folder.
Production build
- Install docker.
- Run
make vmagent-prod
from the root folder of the repository. It buildsvmagent-prod
binary and puts it into thebin
folder.
Building docker images
Run make package-vmagent
. It builds victoriametrics/vmagent:<PKG_TAG>
docker image locally.
<PKG_TAG>
is auto-generated image tag, which depends on source code in the repository.
The <PKG_TAG>
may be manually set via PKG_TAG=foobar make package-vmagent
.