Nikolay
cf1d2f289b
removes FileSize from backup part key ( #1872 )
...
* removes FileSize from backup part key
it should fix download restoration for backups
* Update lib/backup/common/part.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-01 12:30:03 +02:00
Lan
6662714c6c
Add flag of S3ForcePathStyle ( #1802 )
2021-11-17 01:10:22 +02:00
Aliaksandr Valialkin
8dcf814c48
app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS
...
The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls,
while leaving support for `gcs://` urls for backwards compatibility.
2021-09-29 12:12:37 +03:00
Nikolay
6d47e750be
adds check for region with custom s3 endpoint ( #1465 )
2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin
81789da731
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:16:39 +02:00
Aliaksandr Valialkin
de1c07b937
lib/backup: add MustStop() method for all remote filesystems
2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin
bf6d523bef
lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter
2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin
ad0950f630
lib/backup/actions: improve logging to be more clear to humans
2020-10-08 14:23:00 +03:00
Nikolay Khramchikhin
fb356c434b
Changed s3 configProfile flag default, ( #749 )
...
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin
8cc2e01386
lib/backup: allow using ~/.aws/config without region
...
Use us-west-2 for determining bucket region.
2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin
fe77d661b3
all: use errors.As instead of type assertion for detecting net.Error
2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
cf91a94daf
lib/backup: properly create missing parent directories in fs.CreateFile
2020-06-05 19:28:25 +03:00
Aliaksandr Valialkin
8b0d9df51d
lib/backup/fsremote: create all the parent directories before creating file in CreateFile
2020-06-05 10:25:28 +03:00
Aliaksandr Valialkin
0f3d46810b
lib/backup: remove misleading -dst mention in error message
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482
2020-05-15 17:13:27 +03:00
Aliaksandr Valialkin
e72518e8c6
lib/backup: donload only the remaining parts for partially downloaded files after vmrestore restart
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487
2020-05-15 17:03:25 +03:00
Aliaksandr Valialkin
7545784a49
all: fix golangci-lint issues
2020-03-10 19:40:03 +02:00
Aliaksandr Valialkin
7cde594696
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
ea53a21b02
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
1029b6ab34
lib/backup/s3remote: check whether the file exists before deleting it
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284
2020-01-09 23:20:51 +02:00
Aliaksandr Valialkin
705af61587
app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin
7e734433a3
lib/backup: cosmetic fixes after #243
2019-11-29 18:07:41 +02:00
glebsam
4a192cb832
Add option to provide custom endpoint for S3, add option to specify S3 config profile ( #243 )
...
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile
* make fmt
2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin
409c939621
lib/backup: remove flock.lock file in empty dirs
...
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:
1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.
`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin
17eca31989
lib/backup: retrieve only the required metadata when reading GCS objects
2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin
216a260ced
app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin
9d1ee1e2ae
lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore
2019-11-19 18:35:59 +02:00
Oleg Kovalov
74ba42d111
fix misspelled words ( #229 )
2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin
7d7fbf890e
app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-07 21:26:43 +02:00
