Lan
b72eed1f5e
Add flag of S3ForcePathStyle ( #1802 )
2021-11-17 01:03:03 +02:00
Aliaksandr Valialkin
d80d72efec
app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS
...
The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls,
while leaving support for `gcs://` urls for backwards compatibility.
2021-09-29 12:10:29 +03:00
Nikolay
9d45b46f4c
adds check for region with custom s3 endpoint ( #1465 )
2021-07-27 12:35:38 +03:00
Aliaksandr Valialkin
bdd0a1cdb2
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:16:35 +02:00
Aliaksandr Valialkin
b449607181
lib/backup: add MustStop() method for all remote filesystems
2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin
cf5f2874cd
lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter
2020-10-09 15:12:03 +03:00
Aliaksandr Valialkin
9b0a5c1028
lib/backup/actions: improve logging to be more clear to humans
2020-10-08 14:23:07 +03:00
Nikolay Khramchikhin
8d71a60a76
Changed s3 configProfile flag default, ( #749 )
...
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
2020-09-07 21:53:24 +03:00
Aliaksandr Valialkin
a034f02fb2
lib/backup: allow using ~/.aws/config without region
...
Use us-west-2 for determining bucket region.
2020-08-04 13:07:59 +03:00
Aliaksandr Valialkin
56ccfa5218
all: use errors.As instead of type assertion for detecting net.Error
2020-07-01 00:15:34 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
4e8d6b80e0
lib/backup: properly create missing parent directories in fs.CreateFile
2020-06-05 19:28:19 +03:00
Aliaksandr Valialkin
4fd3f6f991
lib/backup/fsremote: create all the parent directories before creating file in CreateFile
2020-06-05 10:25:10 +03:00
Aliaksandr Valialkin
3412d5d138
lib/backup: remove misleading -dst mention in error message
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482
2020-05-15 17:13:37 +03:00
Aliaksandr Valialkin
27f7cca7ff
lib/backup: donload only the remaining parts for partially downloaded files after vmrestore restart
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487
2020-05-15 17:03:33 +03:00
Aliaksandr Valialkin
49d7cb1a3f
all: fix golangci-lint issues
2020-03-10 19:41:46 +02:00
Aliaksandr Valialkin
42864bb52f
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin
680080887d
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
b8038a14e7
lib/backup/s3remote: check whether the file exists before deleting it
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284
2020-01-09 23:20:31 +02:00
Aliaksandr Valialkin
f358fb72d1
app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin
f733cb2186
lib/backup: cosmetic fixes after #243
2019-11-29 18:07:04 +02:00
glebsam
15b7406f7b
Add option to provide custom endpoint for S3, add option to specify S3 config profile ( #243 )
...
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile
* make fmt
2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin
a7125a5b7b
lib/backup: remove flock.lock file in empty dirs
...
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:
1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.
`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
2019-11-28 13:38:58 +02:00
Aliaksandr Valialkin
cfb9fa2100
lib/backup: retrieve only the required metadata when reading GCS objects
2019-11-19 21:06:34 +02:00
Aliaksandr Valialkin
443189fb0a
app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin
2db06f0ef8
lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore
2019-11-19 18:36:23 +02:00
Oleg Kovalov
b4f44befa3
fix misspelled words ( #229 )
2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
1e46961d68
app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-08 21:21:07 +02:00
