From 5edaff516575eb63b0067fbcaddaf92ce4bf53d1 Mon Sep 17 00:00:00 2001 From: Zakhar Bessarab Date: Tue, 29 Nov 2022 22:19:46 +0400 Subject: [PATCH] doc/operator: update formatting for backup section, add FAQ section (#3416) * doc/operator: update formatting for backup section, add FAQ section * doc/operator: address review feedback * doc/operator: add note about difference between `VMRestore` and `VMBackupmanager` as init containers * Update docs/operator/backups.MD Co-authored-by: Roman Khavronenko Co-authored-by: Roman Khavronenko --- docs/operator/FAQ.md | 22 ++++++ docs/operator/backups.MD | 147 +++++++++++++++++++++------------------ 2 files changed, 101 insertions(+), 68 deletions(-) create mode 100644 docs/operator/FAQ.md diff --git a/docs/operator/FAQ.md b/docs/operator/FAQ.md new file mode 100644 index 0000000000..d44c894d98 --- /dev/null +++ b/docs/operator/FAQ.md @@ -0,0 +1,22 @@ +--- +sort: 15 +--- + +# FAQ + +## How to change VMStorage PVC storage class + +With Helm chart deployment: + +1. Update the PVCs manually +2. Run `kubectl delete statefulset --cascade=orphan {vmstorage-sts}` which will delete the sts but keep the pods +3. Update helm chart with the new storage class in the volumeClaimTemplate +4. Run the helm chart to recreate the sts with the updated value + +With Operator deployment: + +1. Update the PVCs manually +2. Run `kubectl delete vmcluster --cascade=orphan {cluster-name}` +3. Run `kubectl delete statefulset --cascade=orphan {vmstorage-sts}` +4. Update VMCluster spec to use new storage class +5. Apply cluster configuration diff --git a/docs/operator/backups.MD b/docs/operator/backups.MD index a5157511d6..18904f08d4 100644 --- a/docs/operator/backups.MD +++ b/docs/operator/backups.MD @@ -6,18 +6,21 @@ sort: 5 ## vmbackupmanager -## vmbackupmanager is proprietary software. +You can check vmbackupmanager [documentation](https://docs.victoriametrics.com/vmbackupmanager.html). It contains a description of the service and its features. This documentation covers vmbackumanager integration in vmoperator - Before using it, you must have signed contract and accept EULA https://victoriametrics.com/assets/VM_EULA.pdf +### vmbackupmanager is a part of VictoriaMetrics Enterprise offer + +*Before using it, you must have signed contract and accept EULA https://victoriametrics.com/assets/VM_EULA.pdf* ## Usage examples - `VMSingle` and `VMCluster` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups. - It supports incremental backups (hours, daily, etc) with popular object storages (aws s3, google cloud storage). - - You can enable it with the simple configuration, define secret - - ```yaml +`VMSingle` and `VMCluster` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups. +It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage). + +The configuration example is the following: + + +```yaml --- apiVersion: v1 kind: Secret @@ -63,74 +66,82 @@ spec: credentialsSecret: name: remote-storage-keys key: credentials +``` -``` -NOTE: for cluster version operator adds suffix for `destination: "s3://your_bucket/folder"`, it becomes `"s3://your_bucket/folder/$(POD_NAME)"`. +NOTE: for cluster version operator adds suffix for destination: `"s3://your_bucket/folder"`, it becomes `"s3://your_bucket/folder/$(POD_NAME)"`. It's needed to make consistent backups for each storage node. - You can read more about backup configuration options and mechanics [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmbackup) - - Possible configuration options for backup crd can be found at [link](https://docs.victoriametrics.com/operator/api.html#vmbackup) +You can read more about backup configuration options and mechanics [here](https://docs.victoriametrics.com/vmbackup.html) + +Possible configuration options for backup crd can be found at [link](https://docs.victoriametrics.com/operator/api.html#vmbackup) ## Restoring backups +There are several ways to restore with [vmrestore](https://docs.victoriametrics.com/vmrestore.html) or [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html). + - It can be done with [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmrestore) - -There two ways: - - First: - You have to stop `VMSingle` by scaling it replicas to zero and manually restore data to the database directory. - - Steps: - 1) edit `VMSingle` CRD, set replicaCount: 0 - 2) wait until database stops - 3) ssh to some server, where you can mount `VMSingle` disk and mount it manually - 4) restore files with `vmrestore` - 5) umount disk - 6) edit `VMSingle` CRD, set replicaCount: 1 - 7) wait database start - - Second: +### Manually mounting disk +You have to stop `VMSingle` by scaling it replicas to zero and manually restore data to the database directory. - 1) add init container with vmrestore command to `VMSingle` CRD, example: -```yaml -apiVersion: operator.victoriametrics.com/v1beta1 -kind: VMSingle -metadata: - name: vmsingle-restored - namespace: monitoring-system -spec: - initContainers: - - name: vmrestore - image: victoriametrics/vmrestore:latest - volumeMounts: - - mountPath: /victoria-metrics-data - name: data - - mountPath: /etc/vm/creds - name: secret-remote-storage-keys - readOnly: true - args: - - -storageDataPath=/victoria-metrics-data - - -src=s3://your_bucket/folder/latest - - -credsFilePath=/etc/vm/creds/credentials - vmBackup: - # This is Enterprise Package feature you need to have signed contract to use it - # and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf - acceptEULA: true - destination: "s3://your_bucket/folder" - extraArgs: - runOnStart: "true" - image: - repository: victoriametrics/vmbackupmanager - tag: v1.67.0-enterprise - credentialsSecret: - name: remote-storage-keys - key: credentials +Steps: +1. edit `VMSingle` CRD, set replicaCount: 0 +2. wait until database stops +3. ssh to some server, where you can mount `VMSingle` disk and mount it manually +4. restore files with `vmrestore` +5. umount disk +6. edit `VMSingle` CRD, set replicaCount: 1 +7. wait database start + +### Using VMRestore init container -``` - 2) apply it, and db will be restored from s3 - - 3) remove initContainers and apply crd. +1. add init container with vmrestore command to `VMSingle` CRD, example: + + ```yaml + apiVersion: operator.victoriametrics.com/v1beta1 + kind: VMSingle + metadata: + name: vmsingle-restored + namespace: monitoring-system + spec: + initContainers: + - name: vmrestore + image: victoriametrics/vmrestore:latest + volumeMounts: + - mountPath: /victoria-metrics-data + name: data + - mountPath: /etc/vm/creds + name: secret-remote-storage-keys + readOnly: true + args: + - -storageDataPath=/victoria-metrics-data + - -src=s3://your_bucket/folder/latest + - -credsFilePath=/etc/vm/creds/credentials + vmBackup: + # This is Enterprise Package feature you need to have signed contract to use it + # and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf + acceptEULA: true + destination: "s3://your_bucket/folder" + extraArgs: + runOnStart: "true" + image: + repository: victoriametrics/vmbackupmanager + tag: v1.83.0-enterprise + credentialsSecret: + name: remote-storage-keys + key: credentials + ``` +2. apply it, and db will be restored from s3 + +3. remove initContainers and apply crd. + +Note that using `VMRestore` will require adjusting `src` for each pod because restore will be handled per-pod. + +### Using VMBackupmanager init container + +Using VMBackupmanager restore in Kubernetes environment is described [here](https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-in-kubernetes). + +Advantages of using `VMBackupmanager` include: +- automatic adjustment of `src` for each pod when backup is requested +- graceful handling of case when no restore is required - `VMBackupmanager` will exit with successful status code and won't prevent pod from starting