Cluster version of VictoriaMetrics

VictoriaMetrics is high-performance and cost-effective time series database. It can be used as a long-term remote storage for Prometheus.

Single-node version is available here.

Prominent features

• Supports all the features of single-node version.
• Scales horizontally to multiple nodes.
• Supports multiple independent namespaces for time series data (aka multi-tenancy).

Architecture overview

VictoriaMetrics cluster consists of the following services:

• vmstorage - stores the data
• vminsert - proxies the ingested data to vmstorage
• vmselect - performs incoming queries using the data from vmstorage

Each service may scale independently and may run on the most suitable hardware.

Building from sources

Development Builds

1. Install go. The minimum supported version is Go 1.12.
2. Run make from the repository root. It should build vmstorage, vmselect and vminsert binaries and put them into the bin folder.

Production builds

There is no need in installing Go on a host system since binaries are built inside the official docker container for Go. This makes reproducible builds. So install docker and run the following command:

make vminsert-prod vmselect-prod vmstorage-prod

Production binaries are built into statically linked binaries for GOARCH=amd64, GOOS=linux. They are put into bin folder with -prod suffixes:

$ make vminsert-prod vmselect-prod vmstorage-prod
$ ls -1 bin
vminsert-prod
vmselect-prod
vmstorage-prod

Building docker images

Run make package. It will build the following docker images locally:

• valyala/vminsert:<PKG_TAG>
• valyala/vmselect:<PKG_TAG>
• valyala/vmstorage:<PKG_TAG>

<PKG_TAG> is auto-generated image tag, which depends on source code in the repository. The <PKG_TAG> may be manually set via PKG_TAG=foobar make package.

Operation

Cluster setup

A minimal cluster must contain the following nodes:

• a single vmstorage node with -retentionPeriod and -storageDataPath flags
• a single vminsert node with -storageNode=<vmstorage_host>:8400
• a single vmselect node with -storageNode=<vmstorage_host>:8401

It is recommended to run at least two nodes for each service for high availability purposes.

An http load balancer must be put in front of vminsert and vmselect nodes:

• requests starting with /insert must be routed to port 8480 on vminsert nodes.
• requests starting with /select must be routed to port 8481 on vmselect nodes.

Ports may be altered by setting -httpListenAddr on the corresponding nodes.

URL format

• URLs for data ingestion: /insert/<accountID>/<suffix>, where:

  • <accountID> is an arbitrary number identifying namespace for data ingestion
  • <suffix> may have the following values:
    • prometheus - for inserting data with Prometheus remote write API
    • influx/write or influx/api/v2/write - for inserting data with Influx line protocol

• URLs for querying: /select/<accountID>/prometheus/<suffix>, where:

  • <accountID> is an arbitrary number identifying data namespace for the query
  • <suffix> may have the following values:
    • api/v1/query - performs PromQL instant query
    • api/v1/query_range - performs PromQL range query
    • api/v1/series - performs series query
    • api/v1/labels - returns a list of label names
    • api/v1/label/<label_name>/values - returns values for the given <label_name> according to API
    • federate - returns federated metrics
    • api/v1/export - exports raw data. See this article for details

• vmstorage nodes provide the following HTTP endpoints on 8482 port:

  • /snapshot/create - create instant snapshot, which can be used for backups in background. Snapshots are created in <storageDataPath>/snapshots folder, where <storageDataPath> is the corresponding command-line flag value.
  • /snapshot/list - list available snasphots.
  • /snapshot/delete?snapshot=<id> - delete the given snapshot.
  • /snapshot/delete_all - delete all the snapshots.

  Snapshots may be created independently on each vmstorage node. There is no need in synchronizing snapshots' creation across vmstorage nodes.

Cluster resizing

• vminsert and vmselect nodes are stateless and may be added / removed at any time. Do not forget updating the list of these nodes on http load balancer.
• vmstorage nodes own the ingested data, so they cannot be removed without data loss.

Steps to add vmstorage node:

1. Start new vmstorage node.
2. Gradually restart all the vmselect nodes with new -storageNode arg containing <new_vmstorage_host>:8401.
3. Gradually restart all the vminsert nodes with new -storageNode arg containing <new_vmstorage_host>:8400.

Cluster availability

• HTTP load balancer must stop routing requests to unavailable vminsert and vmselect nodes.

• The cluster remains available if at least a single vmstorage node exists:

  • vminsert re-routes incoming data from unavailable vmstorage nodes to healthy vmstorage nodes
  • vmselect continues serving partial responses if at least a single vmstorage node is available.

Updating / reconfiguring cluster nodes

All the node types - vminsert, vmselect and vmstorage - may be updated via graceful shutdown. Send SIGINT signal to the corresponding process, wait until it finishes and then start new version with new configs.

Cluster should remain in working state if at least a single node of each type remains available during the update process. See cluster availability section for details.

Helm

• Helm chart is available in the deployment/k8s/helm/victoria-metrics folder.

1. Install Cluster: helm install -n <NAME> deployment/k8s/helm/victoria-mertrics or ENV=<NAME> make helm-install.
2. Upgrade Cluster: helm upgrade <NAME> deployment/k8s/helm/victoria-mertrics or ENV=<NAME> make helm-upgrade.
3. Delete Cluster: helm del --purge <NAME> or ENV=<NAME> make helm-delete.

• Upgrade follows Cluster resizing procedure under the hood.

Community and contributions

We are open to third-party pull requests provided they follow KISS design principle:

• Prefer simple code and architecture.
• Avoid complex abstractions.
• Avoid magic code and fancy algorithms.
• Avoid big external dependencies.
• Minimize the number of moving parts in the distributed system.
• Avoid automated decisions, which may hurt cluster availability, consistency or performance.

Adhering KISS principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.

Due to KISS cluster version of VictoriaMetrics has no the following "features" popular in distributed computing world:

• Fragile gossip protocols.
• Hard-to-understand-and-implement-properly Paxos protocols.
• Complex replication schemes, which may go nuts in unforesseen edge cases. The replication is offloaded to the underlying durable replicated storage such as persistent disks in Google Compute Engine.
• Automatic data reshuffling between storage nodes, which may hurt cluster performance and availability.
• Automatic cluster resizing, which may cost you a lot of money if improperly configured.
• Automatic discovering and addition of new nodes in the cluster, which may mix data between dev and prod clusters :)
• Automatic leader election, which may result in split brain disaster on network errors.

Reporting bugs

Report bugs and propose new features here.

Victoria Metrics Logo

Zip contains three folders with different image orientation (main color and inverted version).

Files included in each folder:

• 2 JPEG Preview files
• 2 PNG Preview files with transparent background
• 2 EPS Adobe Illustrator EPS10 files

Logo Usage Guidelines

Font used:

• Lato Black
• Lato Regular

Color Palette:

• HEX #110f0f
• HEX #ffffff

We kindly ask:

• Please don't use any other font instead of suggested.
• There should be sufficient clear space around the logo.
• Do not change spacing, alignment, or relative locations of the design elements.
• Do not change the proportions of any of the design elements or the design itself. You may resize as needed but must retain all proportions.