diff --git a/app/vmbackupmanager/README.md b/app/vmbackupmanager/README.md index 2394720bb..6c12c5c4c 100644 --- a/app/vmbackupmanager/README.md +++ b/app/vmbackupmanager/README.md @@ -151,6 +151,142 @@ The result on the GCS bucket. We see only 3 daily backups: ![daily](vmbackupmanager_rp_daily_2.png) +## API methods + +`vmbackupmanager` exposes the following API methods: + +* GET `/api/v1/backups` - returns list of backups in remote storage. + Example output: + ```json + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` + +* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands). + Example request body: + ```json + {"backup":"daily/2022-10-06"} + ``` + +* GET `/api/v1/restore` - returns backup name from restore mark if it exists. + Example response: + ```json + {"backup":"daily/2022-10-06"} + ``` + +* DELETE `/api/v1/restore` - delete restore mark. + +## CLI + +`vmbackupmanager` exposes CLI commands to work with [API methods](#api-methods) without external dependencies. + +Supported commands: +```console +vmbackupmanager backup + + vmbackupmanager backup list + List backups in remote storage + +vmbackupmanager restore + Restore backup specified by restore mark if it exists + + vmbackupmanager restore get + Get restore mark if it exists + + vmbackupmanager restore delete + Delete restore mark if it exists + + vmbackupmanager restore create [backup_name] + Create restore mark +``` + +By default, CLI commands are using `http://127.0.0.1:8300` endpoint to reach `vmbackupmanager` API. +It can be changed by using flag: +``` +-apiURL string + vmbackupmanager address to perform API requests (default "http://127.0.0.1:8300") +``` + +### Backup commands + +`vmbackupmanager backup list` lists backups in remote storage: +```console +$ ./vmbackupmanager backup list +["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] +``` + +### Restore commands + +Restore commands are used to create, get and delete restore mark. +Restore mark is used by `vmbackupmanager` to store backup name to restore when running restore. + + +Create restore mark: +```console +$ ./vmbackupmanager restore create daily/2022-10-06 +``` + +Get restore mark if it exists: +```console +$ ./vmbackupmanager restore get +{"backup":"daily/2022-10-06"} +``` + +Delete restore mark if it exists: +```console +$ ./vmbackupmanager restore delete +``` + +Perform restore: +```console +$ /vmbackupmanager-prod restore -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data +``` +Note that `vmsingle` or `vmstorage` should be stopped before performing restore. + +If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vmbackupmanager restore` will exit with successful status code. + +### How to restore backup via CLI + +1. Run `vmbackupmanager backup list` to get list of available backups: + ```console + $ /vmbackupmanager-prod backup list + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` +2. Run `vmbackupmanager restore create` to create restore mark: + - Use relative path to backup to restore from currently used remote storage: + ```console + $ /vmbackupmanager-prod restore create daily/2022-10-06 + ``` + - Use full path to backup to restore from any remote storage: + ```console + $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 + ``` +3. Stop `vmstorage` or `vmsingle` node +4. Run `vmbackupmanager restore` to restore backup: + ```console + $ /vmbackupmanager-prod restore -credsFilePath=credentials.json -storageDataPath=/vmstorage-data + ``` +5. Start `vmstorage` or `vmsingle` node + + +### How to restore in Kubernetes + +1. Enter container running `vmbackupmanager` +2. Use `vmbackupmanager backup list` to get list of available backups: + ```console + $ /vmbackupmanager-prod backup list + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` +3. Use `vmbackupmanager restore create` to create restore mark: + - Use relative path to backup to restore from currently used remote storage: + ```console + $ /vmbackupmanager-prod restore create daily/2022-10-06 + ``` + - Use full path to backup to restore from any remote storage: + ```console + $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 + ``` +4. Restart pod + ## Configuration ### Flags @@ -256,6 +392,8 @@ vmbackupmanager performs regular backups according to the provided configs. -pushmetrics.url array Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. + -restoreOnStart + Check if backup restore was requested and restore requested backup. -runOnStart Upload backups immediately after start of the service. Otherwise the backup starts on new hour -s3ForcePathStyle diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 77b08a759..6bf839bdd 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -56,6 +56,7 @@ The following tip changes can be tested by building VictoriaMetrics components f * FEATURE: log error if some environment variables referred at `-promscrape.config` via `%{ENV_VAR}` aren't found. This should prevent from silent using incorrect config files. * FEATURE: immediately shut down VictoriaMetrics apps on the second SIGINT or SIGTERM signal if they couldn't be finished gracefully for some reason after receiving the first signal. * FEATURE: improve the performance of [/api/v1/series](https://docs.victoriametrics.com/url-examples.html#apiv1series) endpoint by eliminating loading of unused `TSID` data during the API call. +* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): add functionality for automated restore from backup. See [these docs](https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-backup-via-cli). * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly merge buckets with identical `le` values, but with different string representation of these values when calculating [histogram_quantile](https://docs.victoriametrics.com/MetricsQL.html#histogram_quantile) and [histogram_share](https://docs.victoriametrics.com/MetricsQL.html#histogram_share). For example, `http_request_duration_seconds_bucket{le="5"}` and `http_requests_duration_seconds_bucket{le="5.0"}`. Such buckets may be returned from distinct targets. Thanks to @647-coder for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225). * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): change severity level for log messages about failed attempts for sending data to remote storage from `error` to `warn`. The message for about all failed send attempts remains at `error` severity level. diff --git a/docs/vmbackupmanager.md b/docs/vmbackupmanager.md index 05fab1068..9b7a67294 100644 --- a/docs/vmbackupmanager.md +++ b/docs/vmbackupmanager.md @@ -155,6 +155,142 @@ The result on the GCS bucket. We see only 3 daily backups: ![daily](vmbackupmanager_rp_daily_2.png) +## API methods + +`vmbackupmanager` exposes the following API methods: + +* GET `/api/v1/backups` - returns list of backups in remote storage. + Example output: + ```json + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` + +* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands). + Example request body: + ```json + {"backup":"daily/2022-10-06"} + ``` + +* GET `/api/v1/restore` - returns backup name from restore mark if it exists. + Example response: + ```json + {"backup":"daily/2022-10-06"} + ``` + +* DELETE `/api/v1/restore` - delete restore mark. + +## CLI + +`vmbackupmanager` exposes CLI commands to work with [API methods](#api-methods) without external dependencies. + +Supported commands: +```console +vmbackupmanager backup + + vmbackupmanager backup list + List backups in remote storage + +vmbackupmanager restore + Restore backup specified by restore mark if it exists + + vmbackupmanager restore get + Get restore mark if it exists + + vmbackupmanager restore delete + Delete restore mark if it exists + + vmbackupmanager restore create [backup_name] + Create restore mark +``` + +By default, CLI commands are using `http://127.0.0.1:8300` endpoint to reach `vmbackupmanager` API. +It can be changed by using flag: +``` +-apiURL string + vmbackupmanager address to perform API requests (default "http://127.0.0.1:8300") +``` + +### Backup commands + +`vmbackupmanager backup list` lists backups in remote storage: +```console +$ ./vmbackupmanager backup list +["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] +``` + +### Restore commands + +Restore commands are used to create, get and delete restore mark. +Restore mark is used by `vmbackupmanager` to store backup name to restore when running restore. + + +Create restore mark: +```console +$ ./vmbackupmanager restore create daily/2022-10-06 +``` + +Get restore mark if it exists: +```console +$ ./vmbackupmanager restore get +{"backup":"daily/2022-10-06"} +``` + +Delete restore mark if it exists: +```console +$ ./vmbackupmanager restore delete +``` + +Perform restore: +```console +$ /vmbackupmanager-prod restore -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data +``` +Note that `vmsingle` or `vmstorage` should be stopped before performing restore. + +If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vmbackupmanager restore` will exit with successful status code. + +### How to restore backup via CLI + +1. Run `vmbackupmanager backup list` to get list of available backups: + ```console + $ /vmbackupmanager-prod backup list + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` +2. Run `vmbackupmanager restore create` to create restore mark: + - Use relative path to backup to restore from currently used remote storage: + ```console + $ /vmbackupmanager-prod restore create daily/2022-10-06 + ``` + - Use full path to backup to restore from any remote storage: + ```console + $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 + ``` +3. Stop `vmstorage` or `vmsingle` node +4. Run `vmbackupmanager restore` to restore backup: + ```console + $ /vmbackupmanager-prod restore -credsFilePath=credentials.json -storageDataPath=/vmstorage-data + ``` +5. Start `vmstorage` or `vmsingle` node + + +### How to restore in Kubernetes + +1. Enter container running `vmbackupmanager` +2. Use `vmbackupmanager backup list` to get list of available backups: + ```console + $ /vmbackupmanager-prod backup list + ["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] + ``` +3. Use `vmbackupmanager restore create` to create restore mark: + - Use relative path to backup to restore from currently used remote storage: + ```console + $ /vmbackupmanager-prod restore create daily/2022-10-06 + ``` + - Use full path to backup to restore from any remote storage: + ```console + $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 + ``` +4. Restart pod + ## Configuration ### Flags @@ -260,6 +396,8 @@ vmbackupmanager performs regular backups according to the provided configs. -pushmetrics.url array Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. + -restoreOnStart + Check if backup restore was requested and restore requested backup. -runOnStart Upload backups immediately after start of the service. Otherwise the backup starts on new hour -s3ForcePathStyle