mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-23 12:31:07 +01:00
docs/Single-server-VictoriaMetrics.md: make consistent section title sizes
This commit is contained in:
Aliaksandr Valialkin
parent
a6b2b2c005
commit
a11659013f
133
README.md
133
README.md
@ -161,7 +161,7 @@ Click on a link in order to read the corresponding case study
|
||||
* [We kindly ask](#we-kindly-ask)
|
||||
|
||||
|
||||
### How to start VictoriaMetrics
|
||||
## How to start VictoriaMetrics
|
||||
|
||||
Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
|
||||
or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags.
|
||||
@ -178,7 +178,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying-
|
||||
|
||||
It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics.
|
||||
|
||||
#### Environment variables
|
||||
### Environment variables
|
||||
|
||||
Each flag value can be set via environment variables according to these rules:
|
||||
|
||||
@ -188,7 +188,7 @@ Each flag value can be set via environment variables according to these rules:
|
||||
* It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`
|
||||
|
||||
|
||||
### Prometheus setup
|
||||
## Prometheus setup
|
||||
|
||||
Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
|
||||
in order to send data to VictoriaMetrics. Add the following lines
|
||||
@ -244,7 +244,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics
|
||||
which can be used as faster and less resource-hungry alternative to Prometheus in certain cases.
|
||||
|
||||
|
||||
### Grafana setup
|
||||
## Grafana setup
|
||||
|
||||
Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url:
|
||||
|
||||
@ -259,7 +259,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL
|
||||
which is used by Grafana.
|
||||
|
||||
|
||||
### How to upgrade VictoriaMetrics
|
||||
## How to upgrade VictoriaMetrics
|
||||
|
||||
It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
|
||||
say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise.
|
||||
@ -277,7 +277,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
|
||||
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
|
||||
|
||||
|
||||
### How to apply new config to VictoriaMetrics
|
||||
## How to apply new config to VictoriaMetrics
|
||||
|
||||
VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
|
||||
|
||||
@ -289,7 +289,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
|
||||
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
|
||||
|
||||
|
||||
### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
|
||||
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
|
||||
|
||||
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).
|
||||
Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
|
||||
@ -314,7 +314,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](#
|
||||
See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus.
|
||||
|
||||
|
||||
### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
|
||||
## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
|
||||
|
||||
Use `http://<victoriametric-addr>:8428` url instead of InfluxDB url in agents' configs.
|
||||
For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB:
|
||||
@ -376,7 +376,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default](
|
||||
while VictoriaMetrics stores them with *milliseconds* precision.
|
||||
|
||||
|
||||
### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
|
||||
## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
|
||||
|
||||
Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
|
||||
the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`:
|
||||
@ -408,7 +408,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
|
||||
```
|
||||
|
||||
### Querying Graphite data
|
||||
## Querying Graphite data
|
||||
|
||||
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
|
||||
|
||||
@ -417,13 +417,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t
|
||||
* Tags can be explored via [Graphite tags API](#graphite-tags-api-usage)
|
||||
* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
|
||||
|
||||
### How to send data from OpenTSDB-compatible agents
|
||||
## How to send data from OpenTSDB-compatible agents
|
||||
|
||||
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
|
||||
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
|
||||
|
||||
#### Sending data via `telnet put` protocol
|
||||
### Sending data via `telnet put` protocol
|
||||
|
||||
Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance,
|
||||
the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`:
|
||||
@ -453,7 +453,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]}
|
||||
```
|
||||
|
||||
#### Sending OpenTSDB data via HTTP `/api/put` requests
|
||||
### Sending OpenTSDB data via HTTP `/api/put` requests
|
||||
|
||||
Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance,
|
||||
the following command enables OpenTSDB HTTP server on port `4242`:
|
||||
@ -491,7 +491,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
```
|
||||
|
||||
|
||||
### Prometheus querying API usage
|
||||
## Prometheus querying API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/):
|
||||
|
||||
@ -507,7 +507,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
|
||||
|
||||
These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
|
||||
|
||||
#### Prometheus querying API enhancements
|
||||
### Prometheus querying API enhancements
|
||||
|
||||
Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
|
||||
For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
|
||||
@ -529,7 +529,7 @@ Additionally VictoriaMetrics provides the following handlers:
|
||||
* `/api/v1/status/active_queries` - it returns a list of currently running queries.
|
||||
|
||||
|
||||
### Graphite API usage
|
||||
## Graphite API usage
|
||||
|
||||
VictoriaMetrics supports the following Graphite APIs:
|
||||
|
||||
@ -537,7 +537,7 @@ VictoriaMetrics supports the following Graphite APIs:
|
||||
* Tags API - see [these docs](#graphite-tags-api-usage).
|
||||
|
||||
|
||||
#### Graphite Metrics API usage
|
||||
### Graphite Metrics API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
|
||||
|
||||
@ -551,55 +551,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a
|
||||
that start with `node_`. By default `delimiter=.`.
|
||||
|
||||
|
||||
#### Graphite Tags API usage
|
||||
### Graphite Tags API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html):
|
||||
|
||||
* [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
|
||||
* [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
|
||||
* [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/<tag_name>](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
|
||||
* [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
|
||||
|
||||
|
||||
### How to build from sources
|
||||
## How to build from sources
|
||||
|
||||
We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
|
||||
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics
|
||||
from sources. Building from sources is reasonable when developing additional features specific
|
||||
to your needs or when testing bugfixes.
|
||||
|
||||
#### Development build
|
||||
### Development build
|
||||
|
||||
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
|
||||
2. Run `make victoria-metrics` from the root folder of the repository.
|
||||
It builds `victoria-metrics` binary and puts it into the `bin` folder.
|
||||
|
||||
#### Production build
|
||||
### Production build
|
||||
|
||||
1. [Install docker](https://docs.docker.com/install/).
|
||||
2. Run `make victoria-metrics-prod` from the root folder of the repository.
|
||||
It builds `victoria-metrics-prod` binary and puts it into the `bin` folder.
|
||||
|
||||
#### ARM build
|
||||
### ARM build
|
||||
|
||||
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
|
||||
|
||||
#### Development ARM build
|
||||
### Development ARM build
|
||||
|
||||
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
|
||||
2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository.
|
||||
It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder.
|
||||
|
||||
#### Production ARM build
|
||||
### Production ARM build
|
||||
|
||||
1. [Install docker](https://docs.docker.com/install/).
|
||||
2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository.
|
||||
It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder.
|
||||
|
||||
#### Pure Go build (CGO_ENABLED=0)
|
||||
### Pure Go build (CGO_ENABLED=0)
|
||||
|
||||
`Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
|
||||
This is an experimental mode, which may result in a lower compression ratio and slower decompression performance.
|
||||
@ -609,7 +609,7 @@ Use it with caution!
|
||||
2. Run `make victoria-metrics-pure` from the root folder of the repository.
|
||||
It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
|
||||
|
||||
#### Building docker images
|
||||
### Building docker images
|
||||
|
||||
Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:<PKG_TAG>` docker image locally.
|
||||
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
|
||||
@ -623,17 +623,20 @@ For example, the following command builds the image on top of [scratch](https://
|
||||
ROOT_IMAGE=scratch make package-victoria-metrics
|
||||
```
|
||||
|
||||
### Start with docker-compose
|
||||
## Start with docker-compose
|
||||
|
||||
[Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml)
|
||||
helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command.
|
||||
More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker).
|
||||
|
||||
### Setting up service
|
||||
|
||||
## Setting up service
|
||||
|
||||
Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
|
||||
There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics).
|
||||
|
||||
### How to work with snapshots
|
||||
|
||||
## How to work with snapshots
|
||||
|
||||
VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
|
||||
for all the data stored under `-storageDataPath` directory.
|
||||
@ -662,7 +665,7 @@ Steps for restoring from a snapshot:
|
||||
to the directory pointed by `-storageDataPath`.
|
||||
3. Start VictoriaMetrics.
|
||||
|
||||
### How to delete time series
|
||||
## How to delete time series
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
|
||||
where `<timeseries_selector_for_delete>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
@ -694,7 +697,7 @@ It isn't recommended using delete API for the following cases, since it brings n
|
||||
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
|
||||
|
||||
### Forced merge
|
||||
## Forced merge
|
||||
|
||||
VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
|
||||
in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions.
|
||||
@ -709,7 +712,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http
|
||||
when new data is ingested into it.
|
||||
|
||||
|
||||
### How to export time series
|
||||
## How to export time series
|
||||
|
||||
VictoriaMetrics provides the following handlers for exporting data:
|
||||
|
||||
@ -719,7 +722,7 @@ VictoriaMetrics provides the following handlers for exporting data:
|
||||
* `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details.
|
||||
|
||||
|
||||
#### How to export data in native format
|
||||
### How to export data in native format
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
|
||||
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
@ -740,7 +743,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
|
||||
The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
|
||||
|
||||
|
||||
#### How to export data in JSON line format
|
||||
### How to export data in JSON line format
|
||||
|
||||
Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances,
|
||||
since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space.
|
||||
@ -775,7 +778,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search
|
||||
Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format).
|
||||
|
||||
|
||||
#### How to export CSV data
|
||||
### How to export CSV data
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/csv?format=<format>&match=<timeseries_selector_for_export>`,
|
||||
where:
|
||||
@ -799,7 +802,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
|
||||
The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data).
|
||||
|
||||
|
||||
### How to import time series data
|
||||
## How to import time series data
|
||||
|
||||
Time series data can be imported via any supported ingestion protocol:
|
||||
|
||||
@ -816,7 +819,7 @@ Time series data can be imported via any supported ingestion protocol:
|
||||
* `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
|
||||
|
||||
|
||||
#### How to import data in native format
|
||||
### How to import data in native format
|
||||
|
||||
The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`.
|
||||
Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format):
|
||||
@ -845,7 +848,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"`
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import data in JSON line format
|
||||
### How to import data in JSON line format
|
||||
|
||||
Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
|
||||
|
||||
@ -873,7 +876,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import CSV data
|
||||
### How to import CSV data
|
||||
|
||||
Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
|
||||
The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:
|
||||
@ -926,7 +929,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import data in Prometheus exposition format
|
||||
### How to import data in Prometheus exposition format
|
||||
|
||||
VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
|
||||
via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics:
|
||||
@ -961,7 +964,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
|
||||
|
||||
|
||||
|
||||
### Relabeling
|
||||
## Relabeling
|
||||
|
||||
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
|
||||
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
|
||||
@ -985,7 +988,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules:
|
||||
See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling).
|
||||
|
||||
|
||||
### Federation
|
||||
## Federation
|
||||
|
||||
VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/)
|
||||
at `http://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>`.
|
||||
@ -996,7 +999,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series.
|
||||
For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation
|
||||
with scrape intervals exceeding `5m`.
|
||||
|
||||
### Capacity planning
|
||||
## Capacity planning
|
||||
|
||||
A rough estimation of the required resources for ingestion path:
|
||||
|
||||
@ -1036,7 +1039,8 @@ The required resources for query path:
|
||||
* Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually
|
||||
require negligible network bandwidth.
|
||||
|
||||
### High availability
|
||||
|
||||
## High availability
|
||||
|
||||
* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones).
|
||||
* Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag:
|
||||
@ -1077,7 +1081,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair
|
||||
with the enabled de-duplication. See [this section](#deduplication) for details.
|
||||
|
||||
|
||||
### Deduplication
|
||||
## Deduplication
|
||||
|
||||
VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
|
||||
is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
|
||||
@ -1088,7 +1092,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst
|
||||
`external_labels` section in their configs, so they write data to the same time series.
|
||||
|
||||
|
||||
### Retention
|
||||
## Retention
|
||||
|
||||
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
|
||||
that the data will be stored for 3 months and then deleted.
|
||||
@ -1103,7 +1107,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio
|
||||
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
|
||||
|
||||
|
||||
### Multiple retentions
|
||||
## Multiple retentions
|
||||
|
||||
Just start multiple VictoriaMetrics instances with distinct values for the following flags:
|
||||
|
||||
@ -1116,7 +1120,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi
|
||||
The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md).
|
||||
|
||||
|
||||
### Downsampling
|
||||
## Downsampling
|
||||
|
||||
There is no downsampling support at the moment, but:
|
||||
|
||||
@ -1132,11 +1136,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d
|
||||
For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave
|
||||
only a single data point out of 20 initial data points per each 5m interval.
|
||||
|
||||
### Multi-tenancy
|
||||
|
||||
## Multi-tenancy
|
||||
|
||||
Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead.
|
||||
|
||||
### Scalability and cluster version
|
||||
|
||||
## Scalability and cluster version
|
||||
|
||||
Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU.
|
||||
This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions
|
||||
@ -1146,7 +1152,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version]
|
||||
horizontally scalable long-term remote storage for really large Prometheus deployments.
|
||||
[Contact us](mailto:info@victoriametrics.com) for paid support.
|
||||
|
||||
### Alerting
|
||||
|
||||
## Alerting
|
||||
|
||||
It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting.
|
||||
|
||||
@ -1157,7 +1164,7 @@ Additionally, alerting can be set up with the following tools:
|
||||
* With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/).
|
||||
|
||||
|
||||
### Security
|
||||
## Security
|
||||
|
||||
Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet.
|
||||
Consider setting the following command-line flags:
|
||||
@ -1177,7 +1184,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut
|
||||
or similar auth proxy.
|
||||
|
||||
|
||||
### Tuning
|
||||
## Tuning
|
||||
|
||||
* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags,
|
||||
which are automatically adjusted for the available CPU and RAM resources.
|
||||
@ -1193,7 +1200,7 @@ or similar auth proxy.
|
||||
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
## Monitoring
|
||||
|
||||
VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page.
|
||||
These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md)
|
||||
@ -1223,7 +1230,7 @@ The most interesting metrics are:
|
||||
VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page.
|
||||
|
||||
|
||||
### Troubleshooting
|
||||
## Troubleshooting
|
||||
|
||||
* It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need
|
||||
of tweaking these flag values arises.
|
||||
@ -1291,7 +1298,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
|
||||
* VictoriaMetrics ignores `NaN` values during data ingestion.
|
||||
|
||||
|
||||
### Backfilling
|
||||
## Backfilling
|
||||
|
||||
VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data).
|
||||
Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data.
|
||||
@ -1307,7 +1314,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in
|
||||
for data with timestamps close to the current time.
|
||||
|
||||
|
||||
### Data updates
|
||||
## Data updates
|
||||
|
||||
VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
|
||||
for the same time series with identical timestamps. While is possible substituting old time series with new time series via
|
||||
@ -1315,7 +1322,7 @@ for the same time series with identical timestamps. While is possible substituti
|
||||
should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.
|
||||
|
||||
|
||||
### Replication
|
||||
## Replication
|
||||
|
||||
Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead.
|
||||
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details.
|
||||
@ -1325,14 +1332,14 @@ Storage-level replication may be offloaded to durable persistent storage such as
|
||||
See also [high availability docs](#high-availability) and [backup docs](#backups).
|
||||
|
||||
|
||||
### Backups
|
||||
## Backups
|
||||
|
||||
VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md)
|
||||
and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools.
|
||||
We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details.
|
||||
|
||||
|
||||
### Profiling
|
||||
## Profiling
|
||||
|
||||
VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
|
||||
|
||||
@ -1352,6 +1359,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
|
||||
|
||||
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
|
||||
|
||||
|
||||
## Integrations
|
||||
|
||||
* [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts).
|
||||
@ -1365,6 +1373,7 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
|
||||
* [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role).
|
||||
* [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics).
|
||||
|
||||
|
||||
## Third-party contributions
|
||||
|
||||
* [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm))
|
||||
@ -1372,10 +1381,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
|
||||
* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
|
||||
* [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details.
|
||||
|
||||
|
||||
## Contacts
|
||||
|
||||
Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com).
|
||||
|
||||
|
||||
## Community and contributions
|
||||
|
||||
Feel free asking any questions regarding VictoriaMetrics:
|
||||
|
||||
@ -161,7 +161,7 @@ Click on a link in order to read the corresponding case study
|
||||
* [We kindly ask](#we-kindly-ask)
|
||||
|
||||
|
||||
### How to start VictoriaMetrics
|
||||
## How to start VictoriaMetrics
|
||||
|
||||
Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
|
||||
or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags.
|
||||
@ -178,7 +178,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying-
|
||||
|
||||
It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics.
|
||||
|
||||
#### Environment variables
|
||||
### Environment variables
|
||||
|
||||
Each flag value can be set via environment variables according to these rules:
|
||||
|
||||
@ -188,7 +188,7 @@ Each flag value can be set via environment variables according to these rules:
|
||||
* It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`
|
||||
|
||||
|
||||
### Prometheus setup
|
||||
## Prometheus setup
|
||||
|
||||
Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
|
||||
in order to send data to VictoriaMetrics. Add the following lines
|
||||
@ -244,7 +244,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics
|
||||
which can be used as faster and less resource-hungry alternative to Prometheus in certain cases.
|
||||
|
||||
|
||||
### Grafana setup
|
||||
## Grafana setup
|
||||
|
||||
Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url:
|
||||
|
||||
@ -259,7 +259,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL
|
||||
which is used by Grafana.
|
||||
|
||||
|
||||
### How to upgrade VictoriaMetrics
|
||||
## How to upgrade VictoriaMetrics
|
||||
|
||||
It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
|
||||
say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise.
|
||||
@ -277,7 +277,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
|
||||
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
|
||||
|
||||
|
||||
### How to apply new config to VictoriaMetrics
|
||||
## How to apply new config to VictoriaMetrics
|
||||
|
||||
VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
|
||||
|
||||
@ -289,7 +289,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
|
||||
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
|
||||
|
||||
|
||||
### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
|
||||
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
|
||||
|
||||
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).
|
||||
Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
|
||||
@ -314,7 +314,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](#
|
||||
See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus.
|
||||
|
||||
|
||||
### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
|
||||
## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
|
||||
|
||||
Use `http://<victoriametric-addr>:8428` url instead of InfluxDB url in agents' configs.
|
||||
For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB:
|
||||
@ -376,7 +376,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default](
|
||||
while VictoriaMetrics stores them with *milliseconds* precision.
|
||||
|
||||
|
||||
### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
|
||||
## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
|
||||
|
||||
Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
|
||||
the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`:
|
||||
@ -408,7 +408,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
|
||||
```
|
||||
|
||||
### Querying Graphite data
|
||||
## Querying Graphite data
|
||||
|
||||
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
|
||||
|
||||
@ -417,13 +417,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t
|
||||
* Tags can be explored via [Graphite tags API](#graphite-tags-api-usage)
|
||||
* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
|
||||
|
||||
### How to send data from OpenTSDB-compatible agents
|
||||
## How to send data from OpenTSDB-compatible agents
|
||||
|
||||
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
|
||||
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
|
||||
|
||||
#### Sending data via `telnet put` protocol
|
||||
### Sending data via `telnet put` protocol
|
||||
|
||||
Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance,
|
||||
the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`:
|
||||
@ -453,7 +453,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]}
|
||||
```
|
||||
|
||||
#### Sending OpenTSDB data via HTTP `/api/put` requests
|
||||
### Sending OpenTSDB data via HTTP `/api/put` requests
|
||||
|
||||
Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance,
|
||||
the following command enables OpenTSDB HTTP server on port `4242`:
|
||||
@ -491,7 +491,7 @@ The `/api/v1/export` endpoint should return the following response:
|
||||
```
|
||||
|
||||
|
||||
### Prometheus querying API usage
|
||||
## Prometheus querying API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/):
|
||||
|
||||
@ -507,7 +507,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
|
||||
|
||||
These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
|
||||
|
||||
#### Prometheus querying API enhancements
|
||||
### Prometheus querying API enhancements
|
||||
|
||||
Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
|
||||
For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
|
||||
@ -529,7 +529,7 @@ Additionally VictoriaMetrics provides the following handlers:
|
||||
* `/api/v1/status/active_queries` - it returns a list of currently running queries.
|
||||
|
||||
|
||||
### Graphite API usage
|
||||
## Graphite API usage
|
||||
|
||||
VictoriaMetrics supports the following Graphite APIs:
|
||||
|
||||
@ -537,7 +537,7 @@ VictoriaMetrics supports the following Graphite APIs:
|
||||
* Tags API - see [these docs](#graphite-tags-api-usage).
|
||||
|
||||
|
||||
#### Graphite Metrics API usage
|
||||
### Graphite Metrics API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
|
||||
|
||||
@ -551,55 +551,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a
|
||||
that start with `node_`. By default `delimiter=.`.
|
||||
|
||||
|
||||
#### Graphite Tags API usage
|
||||
### Graphite Tags API usage
|
||||
|
||||
VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html):
|
||||
|
||||
* [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
|
||||
* [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
|
||||
* [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/<tag_name>](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
|
||||
* [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
|
||||
* [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
|
||||
|
||||
|
||||
### How to build from sources
|
||||
## How to build from sources
|
||||
|
||||
We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
|
||||
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics
|
||||
from sources. Building from sources is reasonable when developing additional features specific
|
||||
to your needs or when testing bugfixes.
|
||||
|
||||
#### Development build
|
||||
### Development build
|
||||
|
||||
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
|
||||
2. Run `make victoria-metrics` from the root folder of the repository.
|
||||
It builds `victoria-metrics` binary and puts it into the `bin` folder.
|
||||
|
||||
#### Production build
|
||||
### Production build
|
||||
|
||||
1. [Install docker](https://docs.docker.com/install/).
|
||||
2. Run `make victoria-metrics-prod` from the root folder of the repository.
|
||||
It builds `victoria-metrics-prod` binary and puts it into the `bin` folder.
|
||||
|
||||
#### ARM build
|
||||
### ARM build
|
||||
|
||||
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
|
||||
|
||||
#### Development ARM build
|
||||
### Development ARM build
|
||||
|
||||
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
|
||||
2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository.
|
||||
It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder.
|
||||
|
||||
#### Production ARM build
|
||||
### Production ARM build
|
||||
|
||||
1. [Install docker](https://docs.docker.com/install/).
|
||||
2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository.
|
||||
It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder.
|
||||
|
||||
#### Pure Go build (CGO_ENABLED=0)
|
||||
### Pure Go build (CGO_ENABLED=0)
|
||||
|
||||
`Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
|
||||
This is an experimental mode, which may result in a lower compression ratio and slower decompression performance.
|
||||
@ -609,7 +609,7 @@ Use it with caution!
|
||||
2. Run `make victoria-metrics-pure` from the root folder of the repository.
|
||||
It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
|
||||
|
||||
#### Building docker images
|
||||
### Building docker images
|
||||
|
||||
Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:<PKG_TAG>` docker image locally.
|
||||
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
|
||||
@ -623,17 +623,20 @@ For example, the following command builds the image on top of [scratch](https://
|
||||
ROOT_IMAGE=scratch make package-victoria-metrics
|
||||
```
|
||||
|
||||
### Start with docker-compose
|
||||
## Start with docker-compose
|
||||
|
||||
[Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml)
|
||||
helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command.
|
||||
More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker).
|
||||
|
||||
### Setting up service
|
||||
|
||||
## Setting up service
|
||||
|
||||
Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
|
||||
There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics).
|
||||
|
||||
### How to work with snapshots
|
||||
|
||||
## How to work with snapshots
|
||||
|
||||
VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
|
||||
for all the data stored under `-storageDataPath` directory.
|
||||
@ -662,7 +665,7 @@ Steps for restoring from a snapshot:
|
||||
to the directory pointed by `-storageDataPath`.
|
||||
3. Start VictoriaMetrics.
|
||||
|
||||
### How to delete time series
|
||||
## How to delete time series
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
|
||||
where `<timeseries_selector_for_delete>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
@ -694,7 +697,7 @@ It isn't recommended using delete API for the following cases, since it brings n
|
||||
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
|
||||
|
||||
### Forced merge
|
||||
## Forced merge
|
||||
|
||||
VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
|
||||
in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions.
|
||||
@ -709,7 +712,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http
|
||||
when new data is ingested into it.
|
||||
|
||||
|
||||
### How to export time series
|
||||
## How to export time series
|
||||
|
||||
VictoriaMetrics provides the following handlers for exporting data:
|
||||
|
||||
@ -719,7 +722,7 @@ VictoriaMetrics provides the following handlers for exporting data:
|
||||
* `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details.
|
||||
|
||||
|
||||
#### How to export data in native format
|
||||
### How to export data in native format
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
|
||||
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
@ -740,7 +743,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
|
||||
The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
|
||||
|
||||
|
||||
#### How to export data in JSON line format
|
||||
### How to export data in JSON line format
|
||||
|
||||
Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances,
|
||||
since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space.
|
||||
@ -775,7 +778,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search
|
||||
Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format).
|
||||
|
||||
|
||||
#### How to export CSV data
|
||||
### How to export CSV data
|
||||
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/csv?format=<format>&match=<timeseries_selector_for_export>`,
|
||||
where:
|
||||
@ -799,7 +802,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
|
||||
The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data).
|
||||
|
||||
|
||||
### How to import time series data
|
||||
## How to import time series data
|
||||
|
||||
Time series data can be imported via any supported ingestion protocol:
|
||||
|
||||
@ -816,7 +819,7 @@ Time series data can be imported via any supported ingestion protocol:
|
||||
* `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
|
||||
|
||||
|
||||
#### How to import data in native format
|
||||
### How to import data in native format
|
||||
|
||||
The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`.
|
||||
Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format):
|
||||
@ -845,7 +848,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"`
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import data in JSON line format
|
||||
### How to import data in JSON line format
|
||||
|
||||
Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
|
||||
|
||||
@ -873,7 +876,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import CSV data
|
||||
### How to import CSV data
|
||||
|
||||
Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
|
||||
The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:
|
||||
@ -926,7 +929,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la
|
||||
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
|
||||
|
||||
|
||||
#### How to import data in Prometheus exposition format
|
||||
### How to import data in Prometheus exposition format
|
||||
|
||||
VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
|
||||
via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics:
|
||||
@ -961,7 +964,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
|
||||
|
||||
|
||||
|
||||
### Relabeling
|
||||
## Relabeling
|
||||
|
||||
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
|
||||
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
|
||||
@ -985,7 +988,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules:
|
||||
See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling).
|
||||
|
||||
|
||||
### Federation
|
||||
## Federation
|
||||
|
||||
VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/)
|
||||
at `http://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>`.
|
||||
@ -996,7 +999,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series.
|
||||
For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation
|
||||
with scrape intervals exceeding `5m`.
|
||||
|
||||
### Capacity planning
|
||||
## Capacity planning
|
||||
|
||||
A rough estimation of the required resources for ingestion path:
|
||||
|
||||
@ -1036,7 +1039,8 @@ The required resources for query path:
|
||||
* Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually
|
||||
require negligible network bandwidth.
|
||||
|
||||
### High availability
|
||||
|
||||
## High availability
|
||||
|
||||
* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones).
|
||||
* Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag:
|
||||
@ -1077,7 +1081,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair
|
||||
with the enabled de-duplication. See [this section](#deduplication) for details.
|
||||
|
||||
|
||||
### Deduplication
|
||||
## Deduplication
|
||||
|
||||
VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
|
||||
is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
|
||||
@ -1088,7 +1092,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst
|
||||
`external_labels` section in their configs, so they write data to the same time series.
|
||||
|
||||
|
||||
### Retention
|
||||
## Retention
|
||||
|
||||
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
|
||||
that the data will be stored for 3 months and then deleted.
|
||||
@ -1103,7 +1107,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio
|
||||
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
|
||||
|
||||
|
||||
### Multiple retentions
|
||||
## Multiple retentions
|
||||
|
||||
Just start multiple VictoriaMetrics instances with distinct values for the following flags:
|
||||
|
||||
@ -1116,7 +1120,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi
|
||||
The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md).
|
||||
|
||||
|
||||
### Downsampling
|
||||
## Downsampling
|
||||
|
||||
There is no downsampling support at the moment, but:
|
||||
|
||||
@ -1132,11 +1136,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d
|
||||
For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave
|
||||
only a single data point out of 20 initial data points per each 5m interval.
|
||||
|
||||
### Multi-tenancy
|
||||
|
||||
## Multi-tenancy
|
||||
|
||||
Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead.
|
||||
|
||||
### Scalability and cluster version
|
||||
|
||||
## Scalability and cluster version
|
||||
|
||||
Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU.
|
||||
This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions
|
||||
@ -1146,7 +1152,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version]
|
||||
horizontally scalable long-term remote storage for really large Prometheus deployments.
|
||||
[Contact us](mailto:info@victoriametrics.com) for paid support.
|
||||
|
||||
### Alerting
|
||||
|
||||
## Alerting
|
||||
|
||||
It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting.
|
||||
|
||||
@ -1157,7 +1164,7 @@ Additionally, alerting can be set up with the following tools:
|
||||
* With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/).
|
||||
|
||||
|
||||
### Security
|
||||
## Security
|
||||
|
||||
Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet.
|
||||
Consider setting the following command-line flags:
|
||||
@ -1177,7 +1184,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut
|
||||
or similar auth proxy.
|
||||
|
||||
|
||||
### Tuning
|
||||
## Tuning
|
||||
|
||||
* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags,
|
||||
which are automatically adjusted for the available CPU and RAM resources.
|
||||
@ -1193,7 +1200,7 @@ or similar auth proxy.
|
||||
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
## Monitoring
|
||||
|
||||
VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page.
|
||||
These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md)
|
||||
@ -1223,7 +1230,7 @@ The most interesting metrics are:
|
||||
VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page.
|
||||
|
||||
|
||||
### Troubleshooting
|
||||
## Troubleshooting
|
||||
|
||||
* It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need
|
||||
of tweaking these flag values arises.
|
||||
@ -1291,7 +1298,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
|
||||
* VictoriaMetrics ignores `NaN` values during data ingestion.
|
||||
|
||||
|
||||
### Backfilling
|
||||
## Backfilling
|
||||
|
||||
VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data).
|
||||
Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data.
|
||||
@ -1307,7 +1314,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in
|
||||
for data with timestamps close to the current time.
|
||||
|
||||
|
||||
### Data updates
|
||||
## Data updates
|
||||
|
||||
VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
|
||||
for the same time series with identical timestamps. While is possible substituting old time series with new time series via
|
||||
@ -1315,7 +1322,7 @@ for the same time series with identical timestamps. While is possible substituti
|
||||
should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.
|
||||
|
||||
|
||||
### Replication
|
||||
## Replication
|
||||
|
||||
Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead.
|
||||
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details.
|
||||
@ -1325,14 +1332,14 @@ Storage-level replication may be offloaded to durable persistent storage such as
|
||||
See also [high availability docs](#high-availability) and [backup docs](#backups).
|
||||
|
||||
|
||||
### Backups
|
||||
## Backups
|
||||
|
||||
VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md)
|
||||
and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools.
|
||||
We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details.
|
||||
|
||||
|
||||
### Profiling
|
||||
## Profiling
|
||||
|
||||
VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
|
||||
|
||||
@ -1352,6 +1359,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
|
||||
|
||||
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
|
||||
|
||||
|
||||
## Integrations
|
||||
|
||||
* [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts).
|
||||
@ -1363,6 +1371,8 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
|
||||
See [this example](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml).
|
||||
* [Ansible role for installing single-node VictoriaMetrics](https://github.com/dreamteam-gg/ansible-victoriametrics-role).
|
||||
* [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role).
|
||||
* [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics).
|
||||
|
||||
|
||||
## Third-party contributions
|
||||
|
||||
@ -1371,10 +1381,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
|
||||
* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
|
||||
* [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details.
|
||||
|
||||
|
||||
## Contacts
|
||||
|
||||
Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com).
|
||||
|
||||
|
||||
## Community and contributions
|
||||
|
||||
Feel free asking any questions regarding VictoriaMetrics:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user