Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-27 02:46:47 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: increase heading sizes in vmagent, vmauth, vmbackup and vmrestore docs, so they match the heading sizes in VictoriaMetrics docs

Browse Source
This commit is contained in:
Aliaksandr Valialkin 2021-02-01 19:44:00 +02:00
parent b39d5ef656
commit b577cdd855
8 changed files with 108 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

50
app/vmagent/README.md
View File

@ -7,7 +7,7 @@ or any other Prometheus-compatible storage system that supports the `remote_writ
<img alt="vmagent" src="vmagent.png"> <img alt="vmagent" src="vmagent.png">
### Motivation ## Motivation
While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast
and RAM friendly to scrape metrics from Prometheus-compatible exporters to VictoriaMetrics. and RAM friendly to scrape metrics from Prometheus-compatible exporters to VictoriaMetrics.
@ -15,7 +15,7 @@ Also, we found that users’ infrastructure are snowflakes - no two are alike, a
to `vmagent` (like the ability to push metrics instead of pulling them). We did our best and plan to do even more. to `vmagent` (like the ability to push metrics instead of pulling them). We did our best and plan to do even more.
### Features ## Features
* Can be used as drop-in replacement for Prometheus for scraping targets such as [node_exporter](https://github.com/prometheus/node_exporter). * Can be used as drop-in replacement for Prometheus for scraping targets such as [node_exporter](https://github.com/prometheus/node_exporter).
See [Quick Start](#quick-start) for details. See [Quick Start](#quick-start) for details.
@ -36,7 +36,7 @@ to `vmagent` (like the ability to push metrics instead of pulling them). We did
* Uses lower amounts of RAM, CPU, disk IO and network bandwidth compared to Prometheus. * Uses lower amounts of RAM, CPU, disk IO and network bandwidth compared to Prometheus.
### Quick Start ## Quick Start
Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it
and pass the following flags to `vmagent` binary in order to start scraping Prometheus targets: and pass the following flags to `vmagent` binary in order to start scraping Prometheus targets:
@ -63,7 +63,7 @@ Then send Influx data to `http://vmagent-host:8429`. See [these docs](https://vi
Pass `-help` to `vmagent` in order to see the full list of supported command-line flags with their descriptions. Pass `-help` to `vmagent` in order to see the full list of supported command-line flags with their descriptions.
### Configuration update ## Configuration update
`vmagent` should be restarted in order to update config options set via command-line args. `vmagent` should be restarted in order to update config options set via command-line args.
@ -79,10 +79,10 @@ Pass `-help` to `vmagent` in order to see the full list of supported command-lin
There is also `-promscrape.configCheckInterval` command-line option, which can be used for automatic reloading configs from updated `-promscrape.config` file. There is also `-promscrape.configCheckInterval` command-line option, which can be used for automatic reloading configs from updated `-promscrape.config` file.
### Use cases ## Use cases
#### IoT and Edge monitoring ### IoT and Edge monitoring
`vmagent` can run and collect metrics in IoT and industrial networks with unreliable or scheduled connections to the remote storage. `vmagent` can run and collect metrics in IoT and industrial networks with unreliable or scheduled connections to the remote storage.
It buffers the collected data in local files until the connection to remote storage becomes available and then sends the buffered It buffers the collected data in local files until the connection to remote storage becomes available and then sends the buffered
@ -93,14 +93,14 @@ The maximum buffer size can be limited with `-remoteWrite.maxDiskUsagePerURL`.
See [the corresponding Makefile rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/Makefile) for details. See [the corresponding Makefile rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/Makefile) for details.
#### Drop-in replacement for Prometheus ### Drop-in replacement for Prometheus
If you use Prometheus only for scraping metrics from various targets and forwarding these metrics to remote storage, If you use Prometheus only for scraping metrics from various targets and forwarding these metrics to remote storage,
then `vmagent` can replace such Prometheus setup. Usually `vmagent` requires lower amounts of RAM, CPU and network bandwidth comparing to Prometheus for such a setup. then `vmagent` can replace such Prometheus setup. Usually `vmagent` requires lower amounts of RAM, CPU and network bandwidth comparing to Prometheus for such a setup.
See [these docs](#how-to-collect-metrics-in-prometheus-format) for details. See [these docs](#how-to-collect-metrics-in-prometheus-format) for details.
#### Replication and high availability ### Replication and high availability
`vmagent` replicates the collected metrics among multiple remote storage instances configured via `-remoteWrite.url` args. `vmagent` replicates the collected metrics among multiple remote storage instances configured via `-remoteWrite.url` args.
If a single remote storage instance temporarily is out of service, then the collected data remains available in another remote storage instances. If a single remote storage instance temporarily is out of service, then the collected data remains available in another remote storage instances.
@ -108,14 +108,14 @@ If a single remote storage instance temporarily is out of service, then the coll
Then it sends the buffered data to the remote storage in order to prevent data gaps in the remote storage. Then it sends the buffered data to the remote storage in order to prevent data gaps in the remote storage.
#### Relabeling and filtering ### Relabeling and filtering
`vmagent` can add, remove or update labels on the collected data before sending it to remote storage. Additionally, `vmagent` can add, remove or update labels on the collected data before sending it to remote storage. Additionally,
it can remove unwanted samples via Prometheus-like relabeling before sending the collected data to remote storage. it can remove unwanted samples via Prometheus-like relabeling before sending the collected data to remote storage.
See [these docs](#relabeling) for details. See [these docs](#relabeling) for details.
#### Splitting data streams among multiple systems ### Splitting data streams among multiple systems
`vmagent` supports splitting the collected data between muliple destinations with the help of `-remoteWrite.urlRelabelConfig`, `vmagent` supports splitting the collected data between muliple destinations with the help of `-remoteWrite.urlRelabelConfig`,
which is applied independently for each configured `-remoteWrite.url` destination. For instance, it is possible to replicate or split which is applied independently for each configured `-remoteWrite.url` destination. For instance, it is possible to replicate or split
@ -123,7 +123,7 @@ data among long-term remote storage, short-term remote storage and real-time ana
Note that each destination can receive its own subset of the collected data thanks to per-destination relabeling via `-remoteWrite.urlRelabelConfig`. Note that each destination can receive its own subset of the collected data thanks to per-destination relabeling via `-remoteWrite.urlRelabelConfig`.
#### Prometheus remote_write proxy ### Prometheus remote_write proxy
`vmagent` may be used as a proxy for Prometheus data sent via Prometheus `remote_write` protocol. It can accept data via `remote_write` API `vmagent` may be used as a proxy for Prometheus data sent via Prometheus `remote_write` protocol. It can accept data via `remote_write` API
at `/api/v1/write` endpoint, apply relabeling and filtering and then proxy it to another `remote_write` systems. at `/api/v1/write` endpoint, apply relabeling and filtering and then proxy it to another `remote_write` systems.
@ -131,12 +131,12 @@ The `vmagent` can be configured to encrypt the incoming `remote_write` requests
Additionally, Basic Auth can be enabled for the incoming `remote_write` requests with `-httpAuth.*` command-line flags. Additionally, Basic Auth can be enabled for the incoming `remote_write` requests with `-httpAuth.*` command-line flags.
#### remote_write for clustered version ### remote_write for clustered version
Despite `vmagent` can accept data in several supported protocols (OpenTSDB, Influx, Prometheus, Graphite) and scrape data from various targets, writes always peformed in Promethes remote_write protocol. Therefore for clustered version `-remoteWrite.url` command-line flag should be configured as `<schema>://<vminsert-host>:8480/insert/<customer-id>/prometheus/api/v1/write` Despite `vmagent` can accept data in several supported protocols (OpenTSDB, Influx, Prometheus, Graphite) and scrape data from various targets, writes always peformed in Promethes remote_write protocol. Therefore for clustered version `-remoteWrite.url` command-line flag should be configured as `<schema>://<vminsert-host>:8480/insert/<customer-id>/prometheus/api/v1/write`
### How to collect metrics in Prometheus format ## How to collect metrics in Prometheus format
Pass the path to `prometheus.yml` to `-promscrape.config` command-line flag. `vmagent` takes into account the following Pass the path to `prometheus.yml` to `-promscrape.config` command-line flag. `vmagent` takes into account the following
sections from [Prometheus config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/): sections from [Prometheus config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/):
@ -192,7 +192,7 @@ entries to 60s. Run `vmagent -help` in order to see default values for `-promscr
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values.
### Adding labels to metrics ## Adding labels to metrics
Labels can be added to metrics via the following mechanisms: Labels can be added to metrics via the following mechanisms:
@ -200,7 +200,7 @@ Labels can be added to metrics via the following mechanisms:
* Via `-remoteWrite.label` command-line flag. These labels are added to all the collected metrics before sending them to `-remoteWrite.url`. * Via `-remoteWrite.label` command-line flag. These labels are added to all the collected metrics before sending them to `-remoteWrite.url`.
### Relabeling ## Relabeling
`vmagent` supports [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). `vmagent` supports [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).
Additionally it provides the following extra actions: Additionally it provides the following extra actions:
@ -227,7 +227,7 @@ Read more about relabeling in the following articles:
* [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs) * [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
### Monitoring ## Monitoring
`vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page `vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page
either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later. either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later.
@ -246,7 +246,7 @@ This information may be useful for debugging target relabeling.
It may be useful for performing `vmagent` rolling update without scrape loss. It may be useful for performing `vmagent` rolling update without scrape loss.
### Troubleshooting ## Troubleshooting
* It is recommended [setting up the official Grafana dashboard](#monitoring) in order to monitor `vmagent` state. * It is recommended [setting up the official Grafana dashboard](#monitoring) in order to monitor `vmagent` state.
@ -322,24 +322,24 @@ It may be useful for performing `vmagent` rolling update without scrape loss.
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmagent` is located in `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmagent` is located in `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmagent` from the root folder of the repository. 2. Run `make vmagent` from the root folder of the repository.
It builds `vmagent` binary and puts it into the `bin` folder. It builds `vmagent` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmagent-prod` from the root folder of the repository. 2. Run `make vmagent-prod` from the root folder of the repository.
It builds `vmagent-prod` binary and puts it into the `bin` folder. It builds `vmagent-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmagent`. It builds `victoriametrics/vmagent:<PKG_TAG>` docker image locally. Run `make package-vmagent`. It builds `victoriametrics/vmagent:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -352,24 +352,24 @@ by setting it via `<ROOT_IMAGE>` environment variable. For example, the followin
ROOT_IMAGE=scratch make package-vmagent ROOT_IMAGE=scratch make package-vmagent
``` ```
#### ARM build ### ARM build
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/). ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
#### Development ARM build ### Development ARM build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmagent-arm` or `make vmagent-arm64` from the root folder of the repository. 2. Run `make vmagent-arm` or `make vmagent-arm64` from the root folder of the repository.
It builds `vmagent-arm` or `vmagent-arm64` binary respectively and puts it into the `bin` folder. It builds `vmagent-arm` or `vmagent-arm64` binary respectively and puts it into the `bin` folder.
#### Production ARM build ### Production ARM build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmagent-arm-prod` or `make vmagent-arm64-prod` from the root folder of the repository. 2. Run `make vmagent-arm-prod` or `make vmagent-arm64-prod` from the root folder of the repository.
It builds `vmagent-arm-prod` or `vmagent-arm64-prod` binary respectively and puts it into the `bin` folder. It builds `vmagent-arm-prod` or `vmagent-arm64-prod` binary respectively and puts it into the `bin` folder.
### Profiling ## Profiling
`vmagent` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): `vmagent` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):

20
app/vmauth/README.md
View File

@ -5,7 +5,7 @@ It reads username and password from [Basic Auth headers](https://en.wikipedia.or
matches them against configs pointed by `-auth.config` command-line flag and proxies incoming HTTP requests to the configured per-user `url_prefix` on successful match. matches them against configs pointed by `-auth.config` command-line flag and proxies incoming HTTP requests to the configured per-user `url_prefix` on successful match.
### Quick start ## Quick start
Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it
and pass the following flag to `vmauth` binary in order to start authorizing and routing requests: and pass the following flag to `vmauth` binary in order to start authorizing and routing requests:
@ -26,7 +26,7 @@ Pass `-help` to `vmauth` in order to see all the supported command-line flags wi
Feel free [contacting us](mailto:info@victoriametrics.com) if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, limits, etc. Feel free [contacting us](mailto:info@victoriametrics.com) if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, limits, etc.
### Auth config ## Auth config
Auth config is represented in the following simple `yml` format: Auth config is represented in the following simple `yml` format:
@ -68,7 +68,7 @@ The config may contain `%{ENV_VAR}` placeholders, which are substituted by the c
This may be useful for passing secrets to the config. This may be useful for passing secrets to the config.
### Security ## Security
Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable https. This can be done by passing the following `-tls*` command-line flags to `vmauth`: Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable https. This can be done by passing the following `-tls*` command-line flags to `vmauth`:
@ -84,30 +84,30 @@ Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable
Alternatively, [https termination proxy](https://en.wikipedia.org/wiki/TLS_termination_proxy) may be put in front of `vmauth`. Alternatively, [https termination proxy](https://en.wikipedia.org/wiki/TLS_termination_proxy) may be put in front of `vmauth`.
### Monitoring ## Monitoring
`vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page `vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page
either via [vmagent](https://victoriametrics.github.io/vmagent.html) or via Prometheus, so the exported metrics could be analyzed later. either via [vmagent](https://victoriametrics.github.io/vmagent.html) or via Prometheus, so the exported metrics could be analyzed later.
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmauth` is located in `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmauth` is located in `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmauth` from the root folder of the repository. 2. Run `make vmauth` from the root folder of the repository.
It builds `vmauth` binary and puts it into the `bin` folder. It builds `vmauth` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmauth-prod` from the root folder of the repository. 2. Run `make vmauth-prod` from the root folder of the repository.
It builds `vmauth-prod` binary and puts it into the `bin` folder. It builds `vmauth-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmauth`. It builds `victoriametrics/vmauth:<PKG_TAG>` docker image locally. Run `make package-vmauth`. It builds `victoriametrics/vmauth:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -121,7 +121,7 @@ ROOT_IMAGE=scratch make package-vmauth
``` ```
### Profiling ## Profiling
`vmauth` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): `vmauth` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -142,7 +142,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof). The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
### Advanced usage ## Advanced usage
Pass `-help` command-line arg to `vmauth` in order to see all the configuration options: Pass `-help` command-line arg to `vmauth` in order to see all the configuration options:

24
app/vmbackup/README.md
View File

@ -23,9 +23,9 @@ See also [vmbackuper](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/
creation of hourly, daily, weekly and monthly backups. creation of hourly, daily, weekly and monthly backups.
### Use cases ## Use cases
#### Regular backups ### Regular backups
Regular backup can be performed with the following command: Regular backup can be performed with the following command:
@ -40,7 +40,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
* `<path/to/new/backup>` is the destination path where new backup will be placed. * `<path/to/new/backup>` is the destination path where new backup will be placed.
#### Regular backups with server-side copy from existing backup ### Regular backups with server-side copy from existing backup
If the destination GCS bucket already contains the previous backup at `-origin` path, then new backup can be sped up If the destination GCS bucket already contains the previous backup at `-origin` path, then new backup can be sped up
with the following command: with the following command:
@ -52,7 +52,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
It saves time and network bandwidth costs by performing server-side copy for the shared data from the `-origin` to `-dst`. It saves time and network bandwidth costs by performing server-side copy for the shared data from the `-origin` to `-dst`.
#### Incremental backups ### Incremental backups
Incremental backups performed if `-dst` points to an already existing backup. In this case only new data uploaded to remote storage. Incremental backups performed if `-dst` points to an already existing backup. In this case only new data uploaded to remote storage.
It saves time and network bandwidth costs when working with big backups: It saves time and network bandwidth costs when working with big backups:
@ -62,7 +62,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
``` ```
#### Smart backups ### Smart backups
Smart backups mean storing full daily backups into `YYYYMMDD` folders and creating incremental hourly backup into `latest` folder: Smart backups mean storing full daily backups into `YYYYMMDD` folders and creating incremental hourly backup into `latest` folder:
@ -92,7 +92,7 @@ Do not forget removing old snapshots and backups when they are no longer needed
See also [vmbackuper tool](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for automating smart backups. See also [vmbackuper tool](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for automating smart backups.
### How does it work? ## How does it work?
The backup algorithm is the following: The backup algorithm is the following:
@ -118,7 +118,7 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
`vmbackup` can work improperly or slowly when these properties are violated. `vmbackup` can work improperly or slowly when these properties are violated.
### Troubleshooting ## Troubleshooting
* If the backup is slow, then try setting higher value for `-concurrency` flag. This will increase the number of concurrent workers that upload data to backup storage. * If the backup is slow, then try setting higher value for `-concurrency` flag. This will increase the number of concurrent workers that upload data to backup storage.
* If `vmbackup` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value. * If `vmbackup` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
@ -127,7 +127,7 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
at [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) and vice versa. at [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) and vice versa.
### Advanced usage ## Advanced usage
* Obtaining credentials from a file. * Obtaining credentials from a file.
@ -222,24 +222,24 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmbackup` from the root folder of the repository. 2. Run `make vmbackup` from the root folder of the repository.
It builds `vmbackup` binary and puts it into the `bin` folder. It builds `vmbackup` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmbackup-prod` from the root folder of the repository. 2. Run `make vmbackup-prod` from the root folder of the repository.
It builds `vmbackup-prod` binary and puts it into the `bin` folder. It builds `vmbackup-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmbackup`. It builds `victoriametrics/vmbackup:<PKG_TAG>` docker image locally. Run `make package-vmbackup`. It builds `victoriametrics/vmbackup:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.

14
app/vmrestore/README.md
View File

@ -7,7 +7,7 @@ Restore process can be interrupted at any time. It is automatically resumed from
when restarting `vmrestore` with the same args. when restarting `vmrestore` with the same args.
### Usage ## Usage
VictoriaMetrics must be stopped during the restore process. VictoriaMetrics must be stopped during the restore process.
@ -25,13 +25,13 @@ The original `-storageDataPath` directory may contain old files. They will be su
i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/questions/476041/how-do-i-make-rsync-delete-files-that-have-been-deleted-from-the-source-folder). i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/questions/476041/how-do-i-make-rsync-delete-files-that-have-been-deleted-from-the-source-folder).
### Troubleshooting ## Troubleshooting
* If `vmrestore` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value. * If `vmrestore` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
* If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process. * If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process.
### Advanced usage ## Advanced usage
* Obtaining credentials from a file. * Obtaining credentials from a file.
@ -118,24 +118,24 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmrestore` from the root folder of the repository. 2. Run `make vmrestore` from the root folder of the repository.
It builds `vmrestore` binary and puts it into the `bin` folder. It builds `vmrestore` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmrestore-prod` from the root folder of the repository. 2. Run `make vmrestore-prod` from the root folder of the repository.
It builds `vmrestore-prod` binary and puts it into the `bin` folder. It builds `vmrestore-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmrestore`. It builds `victoriametrics/vmrestore:<PKG_TAG>` docker image locally. Run `make package-vmrestore`. It builds `victoriametrics/vmrestore:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.

50
docs/vmagent.md
View File

@ -7,7 +7,7 @@ or any other Prometheus-compatible storage system that supports the `remote_writ
<img alt="vmagent" src="vmagent.png"> <img alt="vmagent" src="vmagent.png">
### Motivation ## Motivation
While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast
and RAM friendly to scrape metrics from Prometheus-compatible exporters to VictoriaMetrics. and RAM friendly to scrape metrics from Prometheus-compatible exporters to VictoriaMetrics.
@ -15,7 +15,7 @@ Also, we found that users’ infrastructure are snowflakes - no two are alike, a
to `vmagent` (like the ability to push metrics instead of pulling them). We did our best and plan to do even more. to `vmagent` (like the ability to push metrics instead of pulling them). We did our best and plan to do even more.
### Features ## Features
* Can be used as drop-in replacement for Prometheus for scraping targets such as [node_exporter](https://github.com/prometheus/node_exporter). * Can be used as drop-in replacement for Prometheus for scraping targets such as [node_exporter](https://github.com/prometheus/node_exporter).
See [Quick Start](#quick-start) for details. See [Quick Start](#quick-start) for details.
@ -36,7 +36,7 @@ to `vmagent` (like the ability to push metrics instead of pulling them). We did
* Uses lower amounts of RAM, CPU, disk IO and network bandwidth compared to Prometheus. * Uses lower amounts of RAM, CPU, disk IO and network bandwidth compared to Prometheus.
### Quick Start ## Quick Start
Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it
and pass the following flags to `vmagent` binary in order to start scraping Prometheus targets: and pass the following flags to `vmagent` binary in order to start scraping Prometheus targets:
@ -63,7 +63,7 @@ Then send Influx data to `http://vmagent-host:8429`. See [these docs](https://vi
Pass `-help` to `vmagent` in order to see the full list of supported command-line flags with their descriptions. Pass `-help` to `vmagent` in order to see the full list of supported command-line flags with their descriptions.
### Configuration update ## Configuration update
`vmagent` should be restarted in order to update config options set via command-line args. `vmagent` should be restarted in order to update config options set via command-line args.
@ -79,10 +79,10 @@ Pass `-help` to `vmagent` in order to see the full list of supported command-lin
There is also `-promscrape.configCheckInterval` command-line option, which can be used for automatic reloading configs from updated `-promscrape.config` file. There is also `-promscrape.configCheckInterval` command-line option, which can be used for automatic reloading configs from updated `-promscrape.config` file.
### Use cases ## Use cases
#### IoT and Edge monitoring ### IoT and Edge monitoring
`vmagent` can run and collect metrics in IoT and industrial networks with unreliable or scheduled connections to the remote storage. `vmagent` can run and collect metrics in IoT and industrial networks with unreliable or scheduled connections to the remote storage.
It buffers the collected data in local files until the connection to remote storage becomes available and then sends the buffered It buffers the collected data in local files until the connection to remote storage becomes available and then sends the buffered
@ -93,14 +93,14 @@ The maximum buffer size can be limited with `-remoteWrite.maxDiskUsagePerURL`.
See [the corresponding Makefile rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/Makefile) for details. See [the corresponding Makefile rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/Makefile) for details.
#### Drop-in replacement for Prometheus ### Drop-in replacement for Prometheus
If you use Prometheus only for scraping metrics from various targets and forwarding these metrics to remote storage, If you use Prometheus only for scraping metrics from various targets and forwarding these metrics to remote storage,
then `vmagent` can replace such Prometheus setup. Usually `vmagent` requires lower amounts of RAM, CPU and network bandwidth comparing to Prometheus for such a setup. then `vmagent` can replace such Prometheus setup. Usually `vmagent` requires lower amounts of RAM, CPU and network bandwidth comparing to Prometheus for such a setup.
See [these docs](#how-to-collect-metrics-in-prometheus-format) for details. See [these docs](#how-to-collect-metrics-in-prometheus-format) for details.
#### Replication and high availability ### Replication and high availability
`vmagent` replicates the collected metrics among multiple remote storage instances configured via `-remoteWrite.url` args. `vmagent` replicates the collected metrics among multiple remote storage instances configured via `-remoteWrite.url` args.
If a single remote storage instance temporarily is out of service, then the collected data remains available in another remote storage instances. If a single remote storage instance temporarily is out of service, then the collected data remains available in another remote storage instances.
@ -108,14 +108,14 @@ If a single remote storage instance temporarily is out of service, then the coll
Then it sends the buffered data to the remote storage in order to prevent data gaps in the remote storage. Then it sends the buffered data to the remote storage in order to prevent data gaps in the remote storage.
#### Relabeling and filtering ### Relabeling and filtering
`vmagent` can add, remove or update labels on the collected data before sending it to remote storage. Additionally, `vmagent` can add, remove or update labels on the collected data before sending it to remote storage. Additionally,
it can remove unwanted samples via Prometheus-like relabeling before sending the collected data to remote storage. it can remove unwanted samples via Prometheus-like relabeling before sending the collected data to remote storage.
See [these docs](#relabeling) for details. See [these docs](#relabeling) for details.
#### Splitting data streams among multiple systems ### Splitting data streams among multiple systems
`vmagent` supports splitting the collected data between muliple destinations with the help of `-remoteWrite.urlRelabelConfig`, `vmagent` supports splitting the collected data between muliple destinations with the help of `-remoteWrite.urlRelabelConfig`,
which is applied independently for each configured `-remoteWrite.url` destination. For instance, it is possible to replicate or split which is applied independently for each configured `-remoteWrite.url` destination. For instance, it is possible to replicate or split
@ -123,7 +123,7 @@ data among long-term remote storage, short-term remote storage and real-time ana
Note that each destination can receive its own subset of the collected data thanks to per-destination relabeling via `-remoteWrite.urlRelabelConfig`. Note that each destination can receive its own subset of the collected data thanks to per-destination relabeling via `-remoteWrite.urlRelabelConfig`.
#### Prometheus remote_write proxy ### Prometheus remote_write proxy
`vmagent` may be used as a proxy for Prometheus data sent via Prometheus `remote_write` protocol. It can accept data via `remote_write` API `vmagent` may be used as a proxy for Prometheus data sent via Prometheus `remote_write` protocol. It can accept data via `remote_write` API
at `/api/v1/write` endpoint, apply relabeling and filtering and then proxy it to another `remote_write` systems. at `/api/v1/write` endpoint, apply relabeling and filtering and then proxy it to another `remote_write` systems.
@ -131,12 +131,12 @@ The `vmagent` can be configured to encrypt the incoming `remote_write` requests
Additionally, Basic Auth can be enabled for the incoming `remote_write` requests with `-httpAuth.*` command-line flags. Additionally, Basic Auth can be enabled for the incoming `remote_write` requests with `-httpAuth.*` command-line flags.
#### remote_write for clustered version ### remote_write for clustered version
Despite `vmagent` can accept data in several supported protocols (OpenTSDB, Influx, Prometheus, Graphite) and scrape data from various targets, writes always peformed in Promethes remote_write protocol. Therefore for clustered version `-remoteWrite.url` command-line flag should be configured as `<schema>://<vminsert-host>:8480/insert/<customer-id>/prometheus/api/v1/write` Despite `vmagent` can accept data in several supported protocols (OpenTSDB, Influx, Prometheus, Graphite) and scrape data from various targets, writes always peformed in Promethes remote_write protocol. Therefore for clustered version `-remoteWrite.url` command-line flag should be configured as `<schema>://<vminsert-host>:8480/insert/<customer-id>/prometheus/api/v1/write`
### How to collect metrics in Prometheus format ## How to collect metrics in Prometheus format
Pass the path to `prometheus.yml` to `-promscrape.config` command-line flag. `vmagent` takes into account the following Pass the path to `prometheus.yml` to `-promscrape.config` command-line flag. `vmagent` takes into account the following
sections from [Prometheus config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/): sections from [Prometheus config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/):
@ -192,7 +192,7 @@ entries to 60s. Run `vmagent -help` in order to see default values for `-promscr
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values.
### Adding labels to metrics ## Adding labels to metrics
Labels can be added to metrics via the following mechanisms: Labels can be added to metrics via the following mechanisms:
@ -200,7 +200,7 @@ Labels can be added to metrics via the following mechanisms:
* Via `-remoteWrite.label` command-line flag. These labels are added to all the collected metrics before sending them to `-remoteWrite.url`. * Via `-remoteWrite.label` command-line flag. These labels are added to all the collected metrics before sending them to `-remoteWrite.url`.
### Relabeling ## Relabeling
`vmagent` supports [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). `vmagent` supports [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).
Additionally it provides the following extra actions: Additionally it provides the following extra actions:
@ -227,7 +227,7 @@ Read more about relabeling in the following articles:
* [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs) * [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
### Monitoring ## Monitoring
`vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page `vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page
either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later. either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later.
@ -246,7 +246,7 @@ This information may be useful for debugging target relabeling.
It may be useful for performing `vmagent` rolling update without scrape loss. It may be useful for performing `vmagent` rolling update without scrape loss.
### Troubleshooting ## Troubleshooting
* It is recommended [setting up the official Grafana dashboard](#monitoring) in order to monitor `vmagent` state. * It is recommended [setting up the official Grafana dashboard](#monitoring) in order to monitor `vmagent` state.
@ -322,24 +322,24 @@ It may be useful for performing `vmagent` rolling update without scrape loss.
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmagent` is located in `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmagent` is located in `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmagent` from the root folder of the repository. 2. Run `make vmagent` from the root folder of the repository.
It builds `vmagent` binary and puts it into the `bin` folder. It builds `vmagent` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmagent-prod` from the root folder of the repository. 2. Run `make vmagent-prod` from the root folder of the repository.
It builds `vmagent-prod` binary and puts it into the `bin` folder. It builds `vmagent-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmagent`. It builds `victoriametrics/vmagent:<PKG_TAG>` docker image locally. Run `make package-vmagent`. It builds `victoriametrics/vmagent:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -352,24 +352,24 @@ by setting it via `<ROOT_IMAGE>` environment variable. For example, the followin
ROOT_IMAGE=scratch make package-vmagent ROOT_IMAGE=scratch make package-vmagent
``` ```
#### ARM build ### ARM build
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/). ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
#### Development ARM build ### Development ARM build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmagent-arm` or `make vmagent-arm64` from the root folder of the repository. 2. Run `make vmagent-arm` or `make vmagent-arm64` from the root folder of the repository.
It builds `vmagent-arm` or `vmagent-arm64` binary respectively and puts it into the `bin` folder. It builds `vmagent-arm` or `vmagent-arm64` binary respectively and puts it into the `bin` folder.
#### Production ARM build ### Production ARM build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmagent-arm-prod` or `make vmagent-arm64-prod` from the root folder of the repository. 2. Run `make vmagent-arm-prod` or `make vmagent-arm64-prod` from the root folder of the repository.
It builds `vmagent-arm-prod` or `vmagent-arm64-prod` binary respectively and puts it into the `bin` folder. It builds `vmagent-arm-prod` or `vmagent-arm64-prod` binary respectively and puts it into the `bin` folder.
### Profiling ## Profiling
`vmagent` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): `vmagent` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):

20
docs/vmauth.md
View File

@ -5,7 +5,7 @@ It reads username and password from [Basic Auth headers](https://en.wikipedia.or
matches them against configs pointed by `-auth.config` command-line flag and proxies incoming HTTP requests to the configured per-user `url_prefix` on successful match. matches them against configs pointed by `-auth.config` command-line flag and proxies incoming HTTP requests to the configured per-user `url_prefix` on successful match.
### Quick start ## Quick start
Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it Just download `vmutils-*` archive from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), unpack it
and pass the following flag to `vmauth` binary in order to start authorizing and routing requests: and pass the following flag to `vmauth` binary in order to start authorizing and routing requests:
@ -26,7 +26,7 @@ Pass `-help` to `vmauth` in order to see all the supported command-line flags wi
Feel free [contacting us](mailto:info@victoriametrics.com) if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, limits, etc. Feel free [contacting us](mailto:info@victoriametrics.com) if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, limits, etc.
### Auth config ## Auth config
Auth config is represented in the following simple `yml` format: Auth config is represented in the following simple `yml` format:
@ -68,7 +68,7 @@ The config may contain `%{ENV_VAR}` placeholders, which are substituted by the c
This may be useful for passing secrets to the config. This may be useful for passing secrets to the config.
### Security ## Security
Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable https. This can be done by passing the following `-tls*` command-line flags to `vmauth`: Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable https. This can be done by passing the following `-tls*` command-line flags to `vmauth`:
@ -84,30 +84,30 @@ Do not transfer Basic Auth headers in plaintext over untrusted networks. Enable
Alternatively, [https termination proxy](https://en.wikipedia.org/wiki/TLS_termination_proxy) may be put in front of `vmauth`. Alternatively, [https termination proxy](https://en.wikipedia.org/wiki/TLS_termination_proxy) may be put in front of `vmauth`.
### Monitoring ## Monitoring
`vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page `vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page
either via [vmagent](https://victoriametrics.github.io/vmagent.html) or via Prometheus, so the exported metrics could be analyzed later. either via [vmagent](https://victoriametrics.github.io/vmagent.html) or via Prometheus, so the exported metrics could be analyzed later.
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmauth` is located in `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - `vmauth` is located in `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmauth` from the root folder of the repository. 2. Run `make vmauth` from the root folder of the repository.
It builds `vmauth` binary and puts it into the `bin` folder. It builds `vmauth` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmauth-prod` from the root folder of the repository. 2. Run `make vmauth-prod` from the root folder of the repository.
It builds `vmauth-prod` binary and puts it into the `bin` folder. It builds `vmauth-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmauth`. It builds `victoriametrics/vmauth:<PKG_TAG>` docker image locally. Run `make package-vmauth`. It builds `victoriametrics/vmauth:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -121,7 +121,7 @@ ROOT_IMAGE=scratch make package-vmauth
``` ```
### Profiling ## Profiling
`vmauth` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs): `vmauth` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -142,7 +142,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof). The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
### Advanced usage ## Advanced usage
Pass `-help` command-line arg to `vmauth` in order to see all the configuration options: Pass `-help` command-line arg to `vmauth` in order to see all the configuration options:

24
docs/vmbackup.md
View File

@ -23,9 +23,9 @@ See also [vmbackuper](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/
creation of hourly, daily, weekly and monthly backups. creation of hourly, daily, weekly and monthly backups.
### Use cases ## Use cases
#### Regular backups ### Regular backups
Regular backup can be performed with the following command: Regular backup can be performed with the following command:
@ -40,7 +40,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
* `<path/to/new/backup>` is the destination path where new backup will be placed. * `<path/to/new/backup>` is the destination path where new backup will be placed.
#### Regular backups with server-side copy from existing backup ### Regular backups with server-side copy from existing backup
If the destination GCS bucket already contains the previous backup at `-origin` path, then new backup can be sped up If the destination GCS bucket already contains the previous backup at `-origin` path, then new backup can be sped up
with the following command: with the following command:
@ -52,7 +52,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
It saves time and network bandwidth costs by performing server-side copy for the shared data from the `-origin` to `-dst`. It saves time and network bandwidth costs by performing server-side copy for the shared data from the `-origin` to `-dst`.
#### Incremental backups ### Incremental backups
Incremental backups performed if `-dst` points to an already existing backup. In this case only new data uploaded to remote storage. Incremental backups performed if `-dst` points to an already existing backup. In this case only new data uploaded to remote storage.
It saves time and network bandwidth costs when working with big backups: It saves time and network bandwidth costs when working with big backups:
@ -62,7 +62,7 @@ vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshotName=<local-
``` ```
#### Smart backups ### Smart backups
Smart backups mean storing full daily backups into `YYYYMMDD` folders and creating incremental hourly backup into `latest` folder: Smart backups mean storing full daily backups into `YYYYMMDD` folders and creating incremental hourly backup into `latest` folder:
@ -92,7 +92,7 @@ Do not forget removing old snapshots and backups when they are no longer needed
See also [vmbackuper tool](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for automating smart backups. See also [vmbackuper tool](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for automating smart backups.
### How does it work? ## How does it work?
The backup algorithm is the following: The backup algorithm is the following:
@ -118,7 +118,7 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
`vmbackup` can work improperly or slowly when these properties are violated. `vmbackup` can work improperly or slowly when these properties are violated.
### Troubleshooting ## Troubleshooting
* If the backup is slow, then try setting higher value for `-concurrency` flag. This will increase the number of concurrent workers that upload data to backup storage. * If the backup is slow, then try setting higher value for `-concurrency` flag. This will increase the number of concurrent workers that upload data to backup storage.
* If `vmbackup` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value. * If `vmbackup` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
@ -127,7 +127,7 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
at [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) and vice versa. at [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) and vice versa.
### Advanced usage ## Advanced usage
* Obtaining credentials from a file. * Obtaining credentials from a file.
@ -222,24 +222,24 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmbackup` from the root folder of the repository. 2. Run `make vmbackup` from the root folder of the repository.
It builds `vmbackup` binary and puts it into the `bin` folder. It builds `vmbackup` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmbackup-prod` from the root folder of the repository. 2. Run `make vmbackup-prod` from the root folder of the repository.
It builds `vmbackup-prod` binary and puts it into the `bin` folder. It builds `vmbackup-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmbackup`. It builds `victoriametrics/vmbackup:<PKG_TAG>` docker image locally. Run `make package-vmbackup`. It builds `victoriametrics/vmbackup:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.

14
docs/vmrestore.md
View File

@ -7,7 +7,7 @@ Restore process can be interrupted at any time. It is automatically resumed from
when restarting `vmrestore` with the same args. when restarting `vmrestore` with the same args.
### Usage ## Usage
VictoriaMetrics must be stopped during the restore process. VictoriaMetrics must be stopped during the restore process.
@ -25,13 +25,13 @@ The original `-storageDataPath` directory may contain old files. They will be su
i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/questions/476041/how-do-i-make-rsync-delete-files-that-have-been-deleted-from-the-source-folder). i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/questions/476041/how-do-i-make-rsync-delete-files-that-have-been-deleted-from-the-source-folder).
### Troubleshooting ## Troubleshooting
* If `vmrestore` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value. * If `vmrestore` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
* If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process. * If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process.
### Advanced usage ## Advanced usage
* Obtaining credentials from a file. * Obtaining credentials from a file.
@ -118,24 +118,24 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
``` ```
### How to build from sources ## How to build from sources
It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there. It is recommended using [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) - see `vmutils-*` archives there.
#### Development build ### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13. 1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make vmrestore` from the root folder of the repository. 2. Run `make vmrestore` from the root folder of the repository.
It builds `vmrestore` binary and puts it into the `bin` folder. It builds `vmrestore` binary and puts it into the `bin` folder.
#### Production build ### Production build
1. [Install docker](https://docs.docker.com/install/). 1. [Install docker](https://docs.docker.com/install/).
2. Run `make vmrestore-prod` from the root folder of the repository. 2. Run `make vmrestore-prod` from the root folder of the repository.
It builds `vmrestore-prod` binary and puts it into the `bin` folder. It builds `vmrestore-prod` binary and puts it into the `bin` folder.
#### Building docker images ### Building docker images
Run `make package-vmrestore`. It builds `victoriametrics/vmrestore:<PKG_TAG>` docker image locally. Run `make package-vmrestore`. It builds `victoriametrics/vmrestore:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.