VictoriaMetrics/lib
justinrush e65e55e2dd
lib/backup: add support for Azure Managed Identity (#6518)
### Describe Your Changes

These changes support using Azure Managed Identity for the `vmbackup`
utility. It adds two new environment variables:

* `AZURE_USE_DEFAULT_CREDENTIAL`: Instructs the `vmbackup` utility to
build a connection using the [Azure Default
Credential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity@v1.5.2#NewDefaultAzureCredential)
mode. This causes the Azure SDK to check for a variety of environment
variables to try and make a connection. By default, it tries to use
managed identity if that is set up.

This will close
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

### Testing

However you normally test the `vmbackup` utility using Azure Blob should
continue to work without any changes. The set up for that is environment
specific and not listed out here.

Once regression testing has been done you can set up [Azure Managed
Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
so your resource (AKS, VM, etc), can use that credential method. Once it
is set up, update your environment variables according to the updated
documentation.

I added unit tests to the `FS.Init` function, then made my changes, then
updated the unit tests to capture the new branches.

I tested this in our environment, but with SAS token auth and managed
identity and it works as expected.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Justin Rush <jarush@epic.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5fd3aef549)
2024-07-10 12:26:21 +02:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:26:02 +02:00
auth lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant 2023-08-30 14:13:51 +02:00
awsapi lib/awsapi: properly assume role with webIdentity token (#5495) 2023-12-20 19:07:04 +02:00
backup lib/backup: add support for Azure Managed Identity (#6518) 2024-07-10 12:26:21 +02:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-07-06 17:22:23 -07:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
cgroup lib/cgroup: remove SetGOGC() function 2024-02-05 12:13:08 +02:00
consts app/vminsert: reduce the max packet size, which vminsert can send to vmstorage 2022-04-05 15:39:58 +03:00
decimal lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
encoding lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:37:05 +02:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 17:57:19 -07:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:24 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
filestream lib/filestream: do not measure read / write duration from / to in-memory buffers 2024-01-23 14:53:35 +02:00
flagutil app/vmagent: add max_scrape_size to scrape config (#6434) 2024-06-20 14:00:22 +02:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:43 -08:00
fs lib/fs/fscore: do not trim content from path (#6503) 2024-06-19 10:37:12 +02:00
handshake lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow 2024-01-23 18:39:28 +02:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:28:17 +01:00
httpserver all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
httputils lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:39:13 +02:00
influxutils lib/flagutil: rename Array to ArrayString 2022-10-01 18:28:19 +03:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:57:49 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 17:02:05 +02:00
logger all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
logstorage app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:55 +02:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
mergeset all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
netutil app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401) 2024-06-12 11:47:44 +02:00
persistentqueue Fixed a typo in the FastQueue mutex comment (#6514) 2024-06-20 14:00:08 +02:00
procutil all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
promauth all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
prompb Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
prompbmarshal Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
promrelabel all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
promscrape all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
promutils all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
protoparser all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
proxy all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-16 21:18:22 +02:00
querytracer all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-04-03 02:41:11 +03:00
regexutil lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
snapshot lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:39:13 +02:00
storage all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
streamaggr all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
stringsutil lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 2023-07-20 21:54:51 -07:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:30:10 +03:00
timeutil all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
uint64set all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
vmselectapi lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00