Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 12:31:07 +01:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity

app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71

Browse Source
This commit is contained in:
Aliaksandr Valialkin 2023-04-13 19:29:06 -07:00
parent 47e16594dd
commit 90b876cd1e
No known key found for this signature in database
GPG Key ID: A72BEC6CD3D0DED1
4 changed files with 28 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
app/vmbackupmanager/README.md
View File

@ -158,8 +158,9 @@ The result on the GCS bucket. We see only 3 daily backups:
* GET `/api/v1/backups` - returns list of backups in remote storage. * GET `/api/v1/backups` - returns list of backups in remote storage.
Example output: Example output:
```json ```json
[{"name":"daily/2022-11-30","size_bytes":26664689,"size":"25.429Mi"},{"name":"daily/2022-12-01","size_bytes":40160965,"size":"38.300Mi"},{"name":"hourly/2022-11-30:12","size_bytes":5846529,"size":"5.576Mi"},{"name":"hourly/2022-11-30:13","size_bytes":17651847,"size":"16.834Mi"},{"name":"hourly/2022-11-30:13:22","size_bytes":8797831,"size":"8.390Mi"},{"name":"hourly/2022-11-30:14","size_bytes":10680454,"size":"10.186Mi"}] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
> Note: `created_at` field is in RFC3339 format.
* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands). * POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands).
Example request body: Example request body:
@ -211,7 +212,7 @@ It can be changed by using flag:
`vmbackupmanager backup list` lists backups in remote storage: `vmbackupmanager backup list` lists backups in remote storage:
```console ```console
$ ./vmbackupmanager backup list $ ./vmbackupmanager backup list
[{"name":"daily/2022-11-30","size_bytes":26664689,"size":"25.429Mi"},{"name":"daily/2022-12-01","size_bytes":40160965,"size":"38.300Mi"},{"name":"hourly/2022-11-30:12","size_bytes":5846529,"size":"5.576Mi"},{"name":"hourly/2022-11-30:13","size_bytes":17651847,"size":"16.834Mi"},{"name":"hourly/2022-11-30:13:22","size_bytes":8797831,"size":"8.390Mi"},{"name":"hourly/2022-11-30:14","size_bytes":10680454,"size":"10.186Mi"}] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
### Restore commands ### Restore commands
@ -249,16 +250,16 @@ If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vm
1. Run `vmbackupmanager backup list` to get list of available backups: 1. Run `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
2. Run `vmbackupmanager restore create` to create restore mark: 2. Run `vmbackupmanager restore create` to create restore mark:
- Use relative path to backup to restore from currently used remote storage: - Use relative path to backup to restore from currently used remote storage:
```console ```console
$ /vmbackupmanager-prod restore create daily/2022-10-06 $ /vmbackupmanager-prod restore create daily/2023-04-07
``` ```
- Use full path to backup to restore from any remote storage: - Use full path to backup to restore from any remote storage:
```console ```console
$ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2023-04-07
``` ```
3. Stop `vmstorage` or `vmsingle` node 3. Stop `vmstorage` or `vmsingle` node
4. Run `vmbackupmanager restore` to restore backup: 4. Run `vmbackupmanager restore` to restore backup:
@ -283,16 +284,16 @@ If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vm
2. Use `vmbackupmanager backup list` to get list of available backups: 2. Use `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
3. Use `vmbackupmanager restore create` to create restore mark: 3. Use `vmbackupmanager restore create` to create restore mark:
- Use relative path to backup to restore from currently used remote storage: - Use relative path to backup to restore from currently used remote storage:
```console ```console
$ /vmbackupmanager-prod restore create daily/2022-10-06 $ /vmbackupmanager-prod restore create daily/2023-04-07
``` ```
- Use full path to backup to restore from any remote storage: - Use full path to backup to restore from any remote storage:
```console ```console
$ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2023-04-07
``` ```
4. Restart pod 4. Restart pod
@ -315,13 +316,13 @@ Clusters here are referred to as `source` and `destination`.
2. Use `vmbackupmanager backup list` to get list of available backups: 2. Use `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
3. Use `vmbackupmanager restore create` to create restore mark at each pod of the *destination* cluster. 3. Use `vmbackupmanager restore create` to create restore mark at each pod of the *destination* cluster.
Each pod in *destination* cluster should be restored from backup of respective pod in *source* cluster. Each pod in *destination* cluster should be restored from backup of respective pod in *source* cluster.
For example: `vmstorage-source-0` in *source* cluster should be restored from `vmstorage-destination-0` in *destination* cluster. For example: `vmstorage-source-0` in *source* cluster should be restored from `vmstorage-destination-0` in *destination* cluster.
```console ```console
$ /vmbackupmanager-prod restore create s3://source_cluster/vmstorage-source-0/daily/2022-10-06 $ /vmbackupmanager-prod restore create s3://source_cluster/vmstorage-source-0/daily/2023-04-07
``` ```
## Monitoring ## Monitoring

2
docs/CHANGELOG.md
View File

@ -16,6 +16,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
## tip ## tip
* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): store backup creation and completion time in `backup_complete.ignore` file of backup contents. This is useful to determine point in time when backup was created and completed. * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): store backup creation and completion time in `backup_complete.ignore` file of backup contents. This is useful to determine point in time when backup was created and completed.
* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): add `created_at` field to the output of `/api/v1/backups` API and `vmbackupmanager backup list` command. See this [doc](https://docs.victoriametrics.com/vmbackupmanager.html#api-methods) for data format details.
## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0) ## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0)
@ -63,6 +64,7 @@ created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or n
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): replace empty labels with `""` instead of `"<no value>"` during templating, as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012). * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): replace empty labels with `""` instead of `"<no value>"` during templating, as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012).
* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): properly pass multiple filters from `--vm-native-filter-match` command-line flag to the data source. Previously filters from `--vm-native-filter-match` were only used to discover the metric names, and the metric names like `__name__="metric_name"` has been taken into account, while the remaining filters were ignored. For example `--vm-native-src-addr={foo="bar",baz="abc"}` may found `metric_name{foo="bar",baz="abc"}` and filter was treated as `--vm-native-src-addr={__name__="metrics_name"}`, e.g. `foo="bar",baz="abc"` filter was ignored. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4062). * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): properly pass multiple filters from `--vm-native-filter-match` command-line flag to the data source. Previously filters from `--vm-native-filter-match` were only used to discover the metric names, and the metric names like `__name__="metric_name"` has been taken into account, while the remaining filters were ignored. For example `--vm-native-src-addr={foo="bar",baz="abc"}` may found `metric_name{foo="bar",baz="abc"}` and filter was treated as `--vm-native-src-addr={__name__="metrics_name"}`, e.g. `foo="bar",baz="abc"` filter was ignored. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4062).
## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1) ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1)
Released at 2023-03-12 Released at 2023-03-12

21
docs/vmbackupmanager.md
View File

@ -162,8 +162,9 @@ The result on the GCS bucket. We see only 3 daily backups:
* GET `/api/v1/backups` - returns list of backups in remote storage. * GET `/api/v1/backups` - returns list of backups in remote storage.
Example output: Example output:
```json ```json
[{"name":"daily/2022-11-30","size_bytes":26664689,"size":"25.429Mi"},{"name":"daily/2022-12-01","size_bytes":40160965,"size":"38.300Mi"},{"name":"hourly/2022-11-30:12","size_bytes":5846529,"size":"5.576Mi"},{"name":"hourly/2022-11-30:13","size_bytes":17651847,"size":"16.834Mi"},{"name":"hourly/2022-11-30:13:22","size_bytes":8797831,"size":"8.390Mi"},{"name":"hourly/2022-11-30:14","size_bytes":10680454,"size":"10.186Mi"}] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
> Note: `created_at` field is in RFC3339 format.
* POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands). * POST `/api/v1/restore` - saves backup name to restore when [performing restore](#restore-commands).
Example request body: Example request body:
@ -215,7 +216,7 @@ It can be changed by using flag:
`vmbackupmanager backup list` lists backups in remote storage: `vmbackupmanager backup list` lists backups in remote storage:
```console ```console
$ ./vmbackupmanager backup list $ ./vmbackupmanager backup list
[{"name":"daily/2022-11-30","size_bytes":26664689,"size":"25.429Mi"},{"name":"daily/2022-12-01","size_bytes":40160965,"size":"38.300Mi"},{"name":"hourly/2022-11-30:12","size_bytes":5846529,"size":"5.576Mi"},{"name":"hourly/2022-11-30:13","size_bytes":17651847,"size":"16.834Mi"},{"name":"hourly/2022-11-30:13:22","size_bytes":8797831,"size":"8.390Mi"},{"name":"hourly/2022-11-30:14","size_bytes":10680454,"size":"10.186Mi"}] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
### Restore commands ### Restore commands
@ -253,16 +254,16 @@ If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vm
1. Run `vmbackupmanager backup list` to get list of available backups: 1. Run `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
2. Run `vmbackupmanager restore create` to create restore mark: 2. Run `vmbackupmanager restore create` to create restore mark:
- Use relative path to backup to restore from currently used remote storage: - Use relative path to backup to restore from currently used remote storage:
```console ```console
$ /vmbackupmanager-prod restore create daily/2022-10-06 $ /vmbackupmanager-prod restore create daily/2023-04-07
``` ```
- Use full path to backup to restore from any remote storage: - Use full path to backup to restore from any remote storage:
```console ```console
$ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2023-04-07
``` ```
3. Stop `vmstorage` or `vmsingle` node 3. Stop `vmstorage` or `vmsingle` node
4. Run `vmbackupmanager restore` to restore backup: 4. Run `vmbackupmanager restore` to restore backup:
@ -287,16 +288,16 @@ If restore mark doesn't exist at `storageDataPath`(restore wasn't requested) `vm
2. Use `vmbackupmanager backup list` to get list of available backups: 2. Use `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
3. Use `vmbackupmanager restore create` to create restore mark: 3. Use `vmbackupmanager restore create` to create restore mark:
- Use relative path to backup to restore from currently used remote storage: - Use relative path to backup to restore from currently used remote storage:
```console ```console
$ /vmbackupmanager-prod restore create daily/2022-10-06 $ /vmbackupmanager-prod restore create daily/2023-04-07
``` ```
- Use full path to backup to restore from any remote storage: - Use full path to backup to restore from any remote storage:
```console ```console
$ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2022-10-06 $ /vmbackupmanager-prod restore create azblob://test1/vmbackupmanager/daily/2023-04-07
``` ```
4. Restart pod 4. Restart pod
@ -319,13 +320,13 @@ Clusters here are referred to as `source` and `destination`.
2. Use `vmbackupmanager backup list` to get list of available backups: 2. Use `vmbackupmanager backup list` to get list of available backups:
```console ```console
$ /vmbackupmanager-prod backup list $ /vmbackupmanager-prod backup list
["daily/2022-10-06","daily/2022-10-10","hourly/2022-10-04:13","hourly/2022-10-06:12","hourly/2022-10-06:13","hourly/2022-10-10:14","hourly/2022-10-10:16","monthly/2022-10","weekly/2022-40","weekly/2022-41"] [{"name":"daily/2023-04-07","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:07+00:00"},{"name":"hourly/2023-04-07:11","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:06+00:00"},{"name":"latest","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:04+00:00"},{"name":"monthly/2023-04","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:10+00:00"},{"name":"weekly/2023-14","size_bytes":318837,"size":"311.4ki","created_at":"2023-04-07T16:15:09+00:00"}]
``` ```
3. Use `vmbackupmanager restore create` to create restore mark at each pod of the *destination* cluster. 3. Use `vmbackupmanager restore create` to create restore mark at each pod of the *destination* cluster.
Each pod in *destination* cluster should be restored from backup of respective pod in *source* cluster. Each pod in *destination* cluster should be restored from backup of respective pod in *source* cluster.
For example: `vmstorage-source-0` in *source* cluster should be restored from `vmstorage-destination-0` in *destination* cluster. For example: `vmstorage-source-0` in *source* cluster should be restored from `vmstorage-destination-0` in *destination* cluster.
```console ```console
$ /vmbackupmanager-prod restore create s3://source_cluster/vmstorage-source-0/daily/2022-10-06 $ /vmbackupmanager-prod restore create s3://source_cluster/vmstorage-source-0/daily/2023-04-07
``` ```
## Monitoring ## Monitoring

7
lib/backup/gcsremote/gcs.go
View File

@ -8,12 +8,13 @@ import (
"time" "time"
"cloud.google.com/go/storage" "cloud.google.com/go/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/common"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/fscommon"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/googleapis/gax-go/v2" "github.com/googleapis/gax-go/v2"
"google.golang.org/api/iterator" "google.golang.org/api/iterator"
"google.golang.org/api/option" "google.golang.org/api/option"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/common"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/fscommon"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
) )
// FS represents filesystem for backups in GCS. // FS represents filesystem for backups in GCS.