VictoriaMetrics/docs/operator/migration.md

---
sort: 5
weight: 5
title: Migration from Prometheus
menu:
  docs:
    parent: "operator"
    weight: 5
aliases:
  - /operator/migration.html
---

# Migration from prometheus-operator

Design and implementation inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator).
It's great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.

So you can use familiar CRD objects: `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe` and `AlertmanagerConfig`.

Or you can use VictoriaMetrics CRDs:

- `VMServiceScrape` (instead of `ServiceMonitor`) - defines scraping metrics configuration from pods backed by services. [See details](./resources/vmservicescrape.md).
- `VMPodScrape` (instead of `PodMonitor`) - defines scraping metrics configuration from pods. [See details](./resources/vmpodscrape.md).
- `VMRule` (instead of `PrometheusRule`) - defines alerting or recording rules. [See details](./resources/vmrule.md).
- `VMProbe` (instead of `Probe`) - defines a probing configuration for targets with blackbox exporter. [See details](./resources/vmprobe.md).
- `VMAlertmanagerConfig` (instead of `AlertmanagerConfig`) - defines a configuration for AlertManager. [See details](./resources/vmalertmanagerconfig.md).
- `VMScrapeConfig` (instead of `ScrapeConfig`) - define a scrape config using any of the service discovery options supported in victoriametrics.

Note that Prometheus CRDs are not supplied with the VictoriaMetrics operator,
so you need to [install them separately](https://github.com/prometheus-operator/prometheus-operator/releases).
VictoriaMetrics operator supports conversion from Prometheus CRD of 
version `monitoring.coreos.com/v1` for kinds `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe` 
and version `monitoring.coreos.com/v1alpha1` for kind `AlertmanagerConfig`.

The default behavior of the operator is as follows:

- It **converts** all existing Prometheus `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe` and `ScrapeConfig` objects into corresponding VictoriaMetrics Operator objects.
- It **syncs** updates (including labels) from Prometheus `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe` and `ScrapeConfig` objects to corresponding VictoriaMetrics Operator objects.
- It **DOES NOT delete** converted objects after original ones are deleted.

With this configuration removing prometheus-operator API objects wouldn't delete any converted objects. So you can safely migrate or run two operators at the same time.

You can change default behavior with operator configuration - [see details below](#objects-conversion).

## Objects conversion

By default, the vmoperator converts all existing [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator)
API objects into corresponding VictoriaMetrics Operator objects ([see above](#migration-from-prometheus-operator)), 
i.e. creates resources of VictoriaMetrics similar to Prometheus resources in the same namespace.

You can control this behaviour by setting env variable for operator:

```sh
# disable convertion for each object
VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR=false
VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROBE=false
VM_ENABLEDPROMETHEUSCONVERTER_SCRAPECONFIG=false
```

For [victoria-metrics-operator helm-chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md) you can use following way:

```yaml
# values.yaml

# ...
operator:
  # -- By default, operator converts prometheus-operator objects.
  disable_prometheus_converter: true
# ...
```

Otherwise, VictoriaMetrics Operator would try to discover prometheus-operator API and convert it.

<img src="migration_prometheus-conversion.png" width="1200">

For more information about the operator's workflow, see [this doc](./README.md).

## Deletion synchronization

By default, the operator doesn't make converted objects disappear after original ones are deleted. To change this behaviour
configure adding `OwnerReferences` to converted objects with following [operator parameter](./setup.md#settings):

```sh
VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true
```

For [victoria-metrics-operator helm-chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md) you can use following way:

```yaml
# values.yaml

# ...
operator:
  # -- Enables ownership reference for converted prometheus-operator objects,
  # it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
  enable_converter_ownership: true
# ...
```

Converted objects will be linked to the original ones and will be deleted by kubernetes after the original ones are deleted.

## Update synchronization

Conversion of api objects can be controlled by annotations, added to `VMObject`s.

Annotation `operator.victoriametrics.com/ignore-prometheus-updates` controls updates from Prometheus api objects.

By default, it set to `disabled`. You define it to `enabled` state and all updates from Prometheus api objects will be ignored.

Example:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    operator.victoriametrics.com/ignore-prometheus-updates: enabled
  labels:
    release: prometheus
  name: prometheus-monitor
spec:
  endpoints: []
```

Annotation `operator.victoriametrics.com/ignore-prometheus-updates` can be set on one of the resources:

- [VMServiceScrape](./resources/vmservicescrape.md)
- [VMPodScrape](./resources/vmpodscrape.md)
- [VMRule](./resources/vmrule.md)
- [VMProbe](./resources/vmprobe.md)
- [VMAlertmanagerConfig](./resources/vmalertmanagerconfig.md)
- [VMScrapeConfig](./resources/vmscrapeconfig.md)

And annotation doesn't make sense for [VMStaticScrape](./resources/vmstaticscrape.md)
and [VMNodeScrape](./resources/vmnodescrape.md) because these objects are not created as a result of conversion.

## Labels and annotations synchronization

Conversion of api objects can be controlled by annotations, added to `VMObject`s.

Annotation `operator.victoriametrics.com/merge-meta-strategy` controls syncing of metadata labels and annotations
between `VMObject`s and `Prometheus` api objects during updates to `Prometheus` objects.

By default, it has `prefer-prometheus`. And annotations and labels will be used from `Prometheus` objects, manually set values will be dropped.

You can set it to `prefer-victoriametrics`. In this case all labels and annotations applied to `Prometheus` object will be ignored and `VMObject` will use own values.

Two additional strategies annotations -`merge-victoriametrics-priority` and `merge-prometheus-priority` merges labelSets into one combined labelSet, with priority.

Example:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
  labels:
    release: prometheus
  name: prometheus-monitor
spec:
  endpoints: []
```

Annotation `operator.victoriametrics.com/merge-meta-strategy` can be set on one of the resources:

- [VMServiceScrape](./resources/vmservicescrape.md)
- [VMPodScrape](./resources/vmpodscrape.md)
- [VMRule](./resources/vmrule.md)
- [VMProbe](./resources/vmprobe.md)
- [VMAlertmanagerConfig](./resources/vmalertmanagerconfig.md)
- [VMScrapeConfig](./resources/vmscrapeconfig.md)

And annotation doesn't make sense for [VMStaticScrape](./resources/vmstaticscrape.md)
and [VMNodeScrape](./resources/vmnodescrape.md) because these objects are not created as a result of conversion.

You can filter labels for syncing 
with [operator parameter](./setup.md#settings) `VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES`:

```sh
# it excludes all labels that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES=helm.sh,argoproj.io
```

In the same way, annotations with specified prefixes can be excluded from synchronization 
with [operator parameter](./setup.md#settings) `VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES`:

```sh
# it excludes all annotations that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES=helm.sh,argoproj.io
```

## Using converter with ArgoCD

If you use ArgoCD, you can allow ignoring objects at ArgoCD converted from Prometheus CRD 
with [operator parameter](./setup.md#settings) `VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS`. 

It helps to properly use converter with ArgoCD and should help prevent out-of-sync issues with argo-cd based deployments:

```sh
# adds compare-options and sync-options for prometheus objects converted by operator 
VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS=true
```

## Data migration

You can use [vmctl](https://docs.victoriametrics.com/vmctl.html) for migrating your data from Prometheus to VictoriaMetrics.

See [this doc](https://docs.victoriametrics.com/vmctl.html#migrating-data-from-prometheus) for more details.

## Auto-discovery for prometheus.io annotations

There is a scenario where auto-discovery using `prometheus.io`-annotations 
(such as `prometheus.io/port`, `prometheus.io/scrape`, `prometheus.io/path`, etc.) 
is required when migrating from Prometheus instead of manually managing scrape objects.

You can enable this feature using special scrape object like that:

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: annotations-discovery
spec:
  discoveryRole: service
  endpoints:
    - port: http
      relabelConfigs:
        # Skip scrape for init containers
        - action: drop
          source_labels: [__meta_kubernetes_pod_container_init]
          regex: "true"
        # Match container port with port from annotation 
        - action: keep_if_equal
          source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
        # Check if scrape is enabled
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: "true"
        # Set scrape path
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        # Set port to address
        - source_labels:
            [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        # Copy labels from pod labels
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        # Set pod name, container name, namespace and node name to labels
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_container_name]
          target_label: container
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_node_name]
          action: replace
          target_label: node
  namespaceSelector: {} # You need to specify namespaceSelector here
  selector: {} # You need to specify selector here
```

You can find yaml-file with this example [here](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmservicescrape_service_sd.yaml).

Check out more information about:
- [VMAgent](./resources/vmagent.md)
- [VMServiceScrape](./resources/vmservicescrape.md)
- [Relabeling](https://docs.victoriametrics.com/vmagent/#relabeling)