From a11659013f70401b9aa425f79938dd6ed241e6cc Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Wed, 18 Nov 2020 12:35:39 +0200 Subject: [PATCH] docs/Single-server-VictoriaMetrics.md: make consistent section title sizes --- README.md | 133 +++++++++++++------------ docs/Single-server-VictoriaMetrics.md | 134 ++++++++++++++------------ 2 files changed, 145 insertions(+), 122 deletions(-) diff --git a/README.md b/README.md index aa9eacfd4..977c9a1d8 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ Click on a link in order to read the corresponding case study * [We kindly ask](#we-kindly-ask) -### How to start VictoriaMetrics +## How to start VictoriaMetrics Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags. @@ -178,7 +178,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying- It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics. -#### Environment variables +### Environment variables Each flag value can be set via environment variables according to these rules: @@ -188,7 +188,7 @@ Each flag value can be set via environment variables according to these rules: * It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_` -### Prometheus setup +## Prometheus setup Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) in order to send data to VictoriaMetrics. Add the following lines @@ -244,7 +244,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics which can be used as faster and less resource-hungry alternative to Prometheus in certain cases. -### Grafana setup +## Grafana setup Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url: @@ -259,7 +259,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL which is used by Grafana. -### How to upgrade VictoriaMetrics +## How to upgrade VictoriaMetrics It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. @@ -277,7 +277,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. -### How to apply new config to VictoriaMetrics +## How to apply new config to VictoriaMetrics VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied: @@ -289,7 +289,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. -### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) +## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. @@ -314,7 +314,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](# See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus. -### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) +## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) Use `http://:8428` url instead of InfluxDB url in agents' configs. For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB: @@ -376,7 +376,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default]( while VictoriaMetrics stores them with *milliseconds* precision. -### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd) +## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd) Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance, the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`: @@ -408,7 +408,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` -### Querying Graphite data +## Querying Graphite data Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs: @@ -417,13 +417,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t * Tags can be explored via [Graphite tags API](#graphite-tags-api-usage) * [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml) -### How to send data from OpenTSDB-compatible agents +## How to send data from OpenTSDB-compatible agents VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html) and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data. The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html). -#### Sending data via `telnet put` protocol +### Sending data via `telnet put` protocol Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance, the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`: @@ -453,7 +453,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]} ``` -#### Sending OpenTSDB data via HTTP `/api/put` requests +### Sending OpenTSDB data via HTTP `/api/put` requests Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance, the following command enables OpenTSDB HTTP server on port `4242`: @@ -491,7 +491,7 @@ The `/api/v1/export` endpoint should return the following response: ``` -### Prometheus querying API usage +## Prometheus querying API usage VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): @@ -507,7 +507,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h These handlers can be queried from Prometheus-compatible clients such as Grafana or curl. -#### Prometheus querying API enhancements +### Prometheus querying API enhancements Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args. For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`. @@ -529,7 +529,7 @@ Additionally VictoriaMetrics provides the following handlers: * `/api/v1/status/active_queries` - it returns a list of currently running queries. -### Graphite API usage +## Graphite API usage VictoriaMetrics supports the following Graphite APIs: @@ -537,7 +537,7 @@ VictoriaMetrics supports the following Graphite APIs: * Tags API - see [these docs](#graphite-tags-api-usage). -#### Graphite Metrics API usage +### Graphite Metrics API usage VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api): @@ -551,55 +551,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a that start with `node_`. By default `delimiter=.`. -#### Graphite Tags API usage +### Graphite Tags API usage VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html): * [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb) * [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb) * [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) -* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) +* [/tags/](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) * [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) * [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support) * [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support) -### How to build from sources +## How to build from sources We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or [docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics from sources. Building from sources is reasonable when developing additional features specific to your needs or when testing bugfixes. -#### Development build +### Development build 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 2. Run `make victoria-metrics` from the root folder of the repository. It builds `victoria-metrics` binary and puts it into the `bin` folder. -#### Production build +### Production build 1. [Install docker](https://docs.docker.com/install/). 2. Run `make victoria-metrics-prod` from the root folder of the repository. It builds `victoria-metrics-prod` binary and puts it into the `bin` folder. -#### ARM build +### ARM build ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/). -#### Development ARM build +### Development ARM build 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository. It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder. -#### Production ARM build +### Production ARM build 1. [Install docker](https://docs.docker.com/install/). 2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository. It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder. -#### Pure Go build (CGO_ENABLED=0) +### Pure Go build (CGO_ENABLED=0) `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies. This is an experimental mode, which may result in a lower compression ratio and slower decompression performance. @@ -609,7 +609,7 @@ Use it with caution! 2. Run `make victoria-metrics-pure` from the root folder of the repository. It builds `victoria-metrics-pure` binary and puts it into the `bin` folder. -#### Building docker images +### Building docker images Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:` docker image locally. `` is auto-generated image tag, which depends on source code in the repository. @@ -623,17 +623,20 @@ For example, the following command builds the image on top of [scratch](https:// ROOT_IMAGE=scratch make package-victoria-metrics ``` -### Start with docker-compose +## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command. More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker). -### Setting up service + +## Setting up service Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS. +There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics). -### How to work with snapshots + +## How to work with snapshots VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) for all the data stored under `-storageDataPath` directory. @@ -662,7 +665,7 @@ Steps for restoring from a snapshot: to the directory pointed by `-storageDataPath`. 3. Start VictoriaMetrics. -### How to delete time series +## How to delete time series Send a request to `http://:8428/api/v1/admin/tsdb/delete_series?match[]=`, where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) @@ -694,7 +697,7 @@ It isn't recommended using delete API for the following cases, since it brings n It is better using `-retentionPeriod` command-line flag for efficient pruning of old data. -### Forced merge +## Forced merge VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions. @@ -709,7 +712,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http when new data is ingested into it. -### How to export time series +## How to export time series VictoriaMetrics provides the following handlers for exporting data: @@ -719,7 +722,7 @@ VictoriaMetrics provides the following handlers for exporting data: * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details. -#### How to export data in native format +### How to export data in native format Send a request to `http://:8428/api/v1/export/native?match[]=`, where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) @@ -740,7 +743,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). -#### How to export data in JSON line format +### How to export data in JSON line format Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances, since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space. @@ -775,7 +778,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). -#### How to export CSV data +### How to export CSV data Send a request to `http://:8428/api/v1/export/csv?format=&match=`, where: @@ -799,7 +802,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). -### How to import time series data +## How to import time series data Time series data can be imported via any supported ingestion protocol: @@ -816,7 +819,7 @@ Time series data can be imported via any supported ingestion protocol: * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details. -#### How to import data in native format +### How to import data in native format The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`. Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format): @@ -845,7 +848,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import data in JSON line format +### How to import data in JSON line format Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format): @@ -873,7 +876,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import CSV data +### How to import CSV data Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg. The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon: @@ -926,7 +929,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import data in Prometheus exposition format +### How to import data in Prometheus exposition format VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics: @@ -961,7 +964,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc -### Relabeling +## Relabeling VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries. @@ -985,7 +988,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules: See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling). -### Federation +## Federation VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/) at `http://:8428/federate?match[]=`. @@ -996,7 +999,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series. For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation with scrape intervals exceeding `5m`. -### Capacity planning +## Capacity planning A rough estimation of the required resources for ingestion path: @@ -1036,7 +1039,8 @@ The required resources for query path: * Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually require negligible network bandwidth. -### High availability + +## High availability * Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). * Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag: @@ -1077,7 +1081,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair with the enabled de-duplication. See [this section](#deduplication) for details. -### Deduplication +## Deduplication VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points @@ -1088,7 +1092,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst `external_labels` section in their configs, so they write data to the same time series. -### Retention +## Retention Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. @@ -1103,7 +1107,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). -### Multiple retentions +## Multiple retentions Just start multiple VictoriaMetrics instances with distinct values for the following flags: @@ -1116,7 +1120,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md). -### Downsampling +## Downsampling There is no downsampling support at the moment, but: @@ -1132,11 +1136,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave only a single data point out of 20 initial data points per each 5m interval. -### Multi-tenancy + +## Multi-tenancy Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead. -### Scalability and cluster version + +## Scalability and cluster version Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU. This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions @@ -1146,7 +1152,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version] horizontally scalable long-term remote storage for really large Prometheus deployments. [Contact us](mailto:info@victoriametrics.com) for paid support. -### Alerting + +## Alerting It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting. @@ -1157,7 +1164,7 @@ Additionally, alerting can be set up with the following tools: * With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/). -### Security +## Security Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet. Consider setting the following command-line flags: @@ -1177,7 +1184,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut or similar auth proxy. -### Tuning +## Tuning * There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags, which are automatically adjusted for the available CPU and RAM resources. @@ -1193,7 +1200,7 @@ or similar auth proxy. mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ``` -### Monitoring +## Monitoring VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page. These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) @@ -1223,7 +1230,7 @@ The most interesting metrics are: VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page. -### Troubleshooting +## Troubleshooting * It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need of tweaking these flag values arises. @@ -1291,7 +1298,7 @@ VictoriaMetrics also exposes currently running queries with their execution time * VictoriaMetrics ignores `NaN` values during data ingestion. -### Backfilling +## Backfilling VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data). Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data. @@ -1307,7 +1314,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in for data with timestamps close to the current time. -### Data updates +## Data updates VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points for the same time series with identical timestamps. While is possible substituting old time series with new time series via @@ -1315,7 +1322,7 @@ for the same time series with identical timestamps. While is possible substituti should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal. -### Replication +## Replication Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead. See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details. @@ -1325,14 +1332,14 @@ Storage-level replication may be offloaded to durable persistent storage such as See also [high availability docs](#high-availability) and [backup docs](#backups). -### Backups +## Backups VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md) and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools. We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details. -### Profiling +## Profiling VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): @@ -1352,6 +1359,7 @@ The command for collecting CPU profile waits for 30 seconds before returning. The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof). + ## Integrations * [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts). @@ -1365,6 +1373,7 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g * [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role). * [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics). + ## Third-party contributions * [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm)) @@ -1372,10 +1381,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g * [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write) * [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details. + ## Contacts Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com). + ## Community and contributions Feel free asking any questions regarding VictoriaMetrics: diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index df5a49b2e..977c9a1d8 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -161,7 +161,7 @@ Click on a link in order to read the corresponding case study * [We kindly ask](#we-kindly-ask) -### How to start VictoriaMetrics +## How to start VictoriaMetrics Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags. @@ -178,7 +178,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying- It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics. -#### Environment variables +### Environment variables Each flag value can be set via environment variables according to these rules: @@ -188,7 +188,7 @@ Each flag value can be set via environment variables according to these rules: * It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_` -### Prometheus setup +## Prometheus setup Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) in order to send data to VictoriaMetrics. Add the following lines @@ -244,7 +244,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics which can be used as faster and less resource-hungry alternative to Prometheus in certain cases. -### Grafana setup +## Grafana setup Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url: @@ -259,7 +259,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL which is used by Grafana. -### How to upgrade VictoriaMetrics +## How to upgrade VictoriaMetrics It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. @@ -277,7 +277,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. -### How to apply new config to VictoriaMetrics +## How to apply new config to VictoriaMetrics VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied: @@ -289,7 +289,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. -### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) +## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. @@ -314,7 +314,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](# See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus. -### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) +## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) Use `http://:8428` url instead of InfluxDB url in agents' configs. For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB: @@ -376,7 +376,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default]( while VictoriaMetrics stores them with *milliseconds* precision. -### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd) +## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd) Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance, the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`: @@ -408,7 +408,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` -### Querying Graphite data +## Querying Graphite data Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs: @@ -417,13 +417,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t * Tags can be explored via [Graphite tags API](#graphite-tags-api-usage) * [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml) -### How to send data from OpenTSDB-compatible agents +## How to send data from OpenTSDB-compatible agents VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html) and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data. The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html). -#### Sending data via `telnet put` protocol +### Sending data via `telnet put` protocol Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance, the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`: @@ -453,7 +453,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]} ``` -#### Sending OpenTSDB data via HTTP `/api/put` requests +### Sending OpenTSDB data via HTTP `/api/put` requests Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance, the following command enables OpenTSDB HTTP server on port `4242`: @@ -491,7 +491,7 @@ The `/api/v1/export` endpoint should return the following response: ``` -### Prometheus querying API usage +## Prometheus querying API usage VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): @@ -507,7 +507,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h These handlers can be queried from Prometheus-compatible clients such as Grafana or curl. -#### Prometheus querying API enhancements +### Prometheus querying API enhancements Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args. For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`. @@ -529,7 +529,7 @@ Additionally VictoriaMetrics provides the following handlers: * `/api/v1/status/active_queries` - it returns a list of currently running queries. -### Graphite API usage +## Graphite API usage VictoriaMetrics supports the following Graphite APIs: @@ -537,7 +537,7 @@ VictoriaMetrics supports the following Graphite APIs: * Tags API - see [these docs](#graphite-tags-api-usage). -#### Graphite Metrics API usage +### Graphite Metrics API usage VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api): @@ -551,55 +551,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a that start with `node_`. By default `delimiter=.`. -#### Graphite Tags API usage +### Graphite Tags API usage VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html): * [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb) * [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb) * [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) -* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) +* [/tags/](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) * [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags) * [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support) * [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support) -### How to build from sources +## How to build from sources We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or [docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics from sources. Building from sources is reasonable when developing additional features specific to your needs or when testing bugfixes. -#### Development build +### Development build 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 2. Run `make victoria-metrics` from the root folder of the repository. It builds `victoria-metrics` binary and puts it into the `bin` folder. -#### Production build +### Production build 1. [Install docker](https://docs.docker.com/install/). 2. Run `make victoria-metrics-prod` from the root folder of the repository. It builds `victoria-metrics-prod` binary and puts it into the `bin` folder. -#### ARM build +### ARM build ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/). -#### Development ARM build +### Development ARM build 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository. It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder. -#### Production ARM build +### Production ARM build 1. [Install docker](https://docs.docker.com/install/). 2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository. It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder. -#### Pure Go build (CGO_ENABLED=0) +### Pure Go build (CGO_ENABLED=0) `Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies. This is an experimental mode, which may result in a lower compression ratio and slower decompression performance. @@ -609,7 +609,7 @@ Use it with caution! 2. Run `make victoria-metrics-pure` from the root folder of the repository. It builds `victoria-metrics-pure` binary and puts it into the `bin` folder. -#### Building docker images +### Building docker images Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:` docker image locally. `` is auto-generated image tag, which depends on source code in the repository. @@ -623,17 +623,20 @@ For example, the following command builds the image on top of [scratch](https:// ROOT_IMAGE=scratch make package-victoria-metrics ``` -### Start with docker-compose +## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command. More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker). -### Setting up service + +## Setting up service Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS. +There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics). -### How to work with snapshots + +## How to work with snapshots VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) for all the data stored under `-storageDataPath` directory. @@ -662,7 +665,7 @@ Steps for restoring from a snapshot: to the directory pointed by `-storageDataPath`. 3. Start VictoriaMetrics. -### How to delete time series +## How to delete time series Send a request to `http://:8428/api/v1/admin/tsdb/delete_series?match[]=`, where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) @@ -694,7 +697,7 @@ It isn't recommended using delete API for the following cases, since it brings n It is better using `-retentionPeriod` command-line flag for efficient pruning of old data. -### Forced merge +## Forced merge VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions. @@ -709,7 +712,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http when new data is ingested into it. -### How to export time series +## How to export time series VictoriaMetrics provides the following handlers for exporting data: @@ -719,7 +722,7 @@ VictoriaMetrics provides the following handlers for exporting data: * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details. -#### How to export data in native format +### How to export data in native format Send a request to `http://:8428/api/v1/export/native?match[]=`, where `` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) @@ -740,7 +743,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format). -#### How to export data in JSON line format +### How to export data in JSON line format Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances, since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space. @@ -775,7 +778,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). -#### How to export CSV data +### How to export CSV data Send a request to `http://:8428/api/v1/export/csv?format=&match=`, where: @@ -799,7 +802,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data). -### How to import time series data +## How to import time series data Time series data can be imported via any supported ingestion protocol: @@ -816,7 +819,7 @@ Time series data can be imported via any supported ingestion protocol: * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details. -#### How to import data in native format +### How to import data in native format The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`. Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format): @@ -845,7 +848,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import data in JSON line format +### How to import data in JSON line format Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format): @@ -873,7 +876,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import CSV data +### How to import CSV data Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg. The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon: @@ -926,7 +929,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. -#### How to import data in Prometheus exposition format +### How to import data in Prometheus exposition format VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics: @@ -961,7 +964,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc -### Relabeling +## Relabeling VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries. @@ -985,7 +988,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules: See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling). -### Federation +## Federation VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/) at `http://:8428/federate?match[]=`. @@ -996,7 +999,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series. For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation with scrape intervals exceeding `5m`. -### Capacity planning +## Capacity planning A rough estimation of the required resources for ingestion path: @@ -1036,7 +1039,8 @@ The required resources for query path: * Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually require negligible network bandwidth. -### High availability + +## High availability * Install multiple VictoriaMetrics instances in distinct datacenters (availability zones). * Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag: @@ -1077,7 +1081,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair with the enabled de-duplication. See [this section](#deduplication) for details. -### Deduplication +## Deduplication VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points @@ -1088,7 +1092,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst `external_labels` section in their configs, so they write data to the same time series. -### Retention +## Retention Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. @@ -1103,7 +1107,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). -### Multiple retentions +## Multiple retentions Just start multiple VictoriaMetrics instances with distinct values for the following flags: @@ -1116,7 +1120,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md). -### Downsampling +## Downsampling There is no downsampling support at the moment, but: @@ -1132,11 +1136,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave only a single data point out of 20 initial data points per each 5m interval. -### Multi-tenancy + +## Multi-tenancy Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead. -### Scalability and cluster version + +## Scalability and cluster version Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU. This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions @@ -1146,7 +1152,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version] horizontally scalable long-term remote storage for really large Prometheus deployments. [Contact us](mailto:info@victoriametrics.com) for paid support. -### Alerting + +## Alerting It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting. @@ -1157,7 +1164,7 @@ Additionally, alerting can be set up with the following tools: * With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/). -### Security +## Security Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet. Consider setting the following command-line flags: @@ -1177,7 +1184,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut or similar auth proxy. -### Tuning +## Tuning * There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags, which are automatically adjusted for the available CPU and RAM resources. @@ -1193,7 +1200,7 @@ or similar auth proxy. mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ``` -### Monitoring +## Monitoring VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page. These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) @@ -1223,7 +1230,7 @@ The most interesting metrics are: VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page. -### Troubleshooting +## Troubleshooting * It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need of tweaking these flag values arises. @@ -1291,7 +1298,7 @@ VictoriaMetrics also exposes currently running queries with their execution time * VictoriaMetrics ignores `NaN` values during data ingestion. -### Backfilling +## Backfilling VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data). Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data. @@ -1307,7 +1314,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in for data with timestamps close to the current time. -### Data updates +## Data updates VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points for the same time series with identical timestamps. While is possible substituting old time series with new time series via @@ -1315,7 +1322,7 @@ for the same time series with identical timestamps. While is possible substituti should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal. -### Replication +## Replication Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead. See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details. @@ -1325,14 +1332,14 @@ Storage-level replication may be offloaded to durable persistent storage such as See also [high availability docs](#high-availability) and [backup docs](#backups). -### Backups +## Backups VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md) and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools. We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details. -### Profiling +## Profiling VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): @@ -1352,6 +1359,7 @@ The command for collecting CPU profile waits for 30 seconds before returning. The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof). + ## Integrations * [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts). @@ -1363,6 +1371,8 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g See [this example](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml). * [Ansible role for installing single-node VictoriaMetrics](https://github.com/dreamteam-gg/ansible-victoriametrics-role). * [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role). +* [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics). + ## Third-party contributions @@ -1371,10 +1381,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g * [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write) * [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details. + ## Contacts Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com). + ## Community and contributions Feel free asking any questions regarding VictoriaMetrics: