Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 08:23:34 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
803ab6ed30
VictoriaMetrics/docs/guides/guide-vmcluster-multiple-retention-setup.md
Artem Navoiev b030662e3e
update multi-tenancy guide. Add infromation about Enterprise and Rete… (#3285) 
docs: update multi-tenancy guide

Add information about Enterprise and Retention Filters

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-11-05 10:09:11 +02:00

3.8 KiB
Raw Blame History

Multi Retention Setup within VictoriaMetrics Cluster

Objective

Setup Victoria Metrics Cluster with support of multiple retention periods within one installation.

Challenge

If you use VictoriaMetrics enterprise, then you can use retention filters for applying multiple retentions to distinct sets of time series and/or tenants.

Community version of VictoriaMetrics supports only one retention period via -retentionPeriod command-line flag. The following solution allows supporting multiple retentions for community version of VictoriaMetrics.

Enterprise Solution

Enterprise version of VictoriaMetrics Single and Cluster support multi-retention from version 1.83.0 via -retentionFilters command-line flag. For example:

  • VictoriaMetrics Single - -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y
  • VictoriaMetrics Cluster - -retentionFilter='{vm_account_id=~"42.*"}:1d' -retentionFilter='{env=~"dev|staging"}:3d' -retentionPeriod=4w. Labels vm_account_id and vm_project_id can be used to specify tenant and project respectively.

Multi-retention is explained in more detail in corresponding docs for single-node and cluster versions.

Open Source Solution

A multi-retention setup can be implemented by dividing a victoriametrics cluster into logical groups with different retentions.

Example: Setup should handle 3 different retention groups 3months, 1year and 3 years. Solution contains 3 groups of vmstorages + vminserst and one group of vmselects. Routing is done by vmagent and relabeling configuration. The -retentionPeriod sets how long to keep the metrics.

The diagram below shows a proposed solution

Implementation Details

  1. Groups of vminserts A know about only vmstorages A and this is explicitly specified via -storageNode configuration.
  2. Groups of vminserts B know about only vmstorages B and this is explicitly specified via -storageNode configuration.
  3. Groups of vminserts C know about only vmstorages A and this is explicitly specified via -storageNode configuration.
  4. Vmselect reads data from all vmstorage nodes via -storageNode configuration.
  5. Vmagent routes incoming metrics to the given set of vminsert nodes using relabeling rules specified at -remoteWrite.urlRelabelConfig configuration.

Multi-Tenant Setup

Every group of vmstorages can handle one tenant or multiple one. Different groups can have overlapping tenants. As vmselect reads from all vmstorage nodes, the data is aggregated on its level.

Additional Enhancements

You can set up vmauth for routing data to the given vminsert group depending on the needed retention.