`VMAgent` represents agent, which helps you collect metrics from various sources and stores them in VictoriaMetrics.
The `VMAgent` CRD declaratively defines a desired [VMAgent](https://docs.victoriametrics.com/vmagent)
setup to run in a Kubernetes cluster.
It requires access to Kubernetes API and you can create RBAC for it first, it can be found
at [`examples/vmagent_rbac.yaml`](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmagent_rbac.yaml)
Or you can use default rbac account, that will be created for `VMAgent` by operator automatically.
For each `VMAgent` resource Operator deploys a properly configured `Deployment` in the same namespace.
The VMAgent `Pod`s are configured to mount a `Secret` prefixed with `<VMAgent-name>` containing the configuration
for VMAgent.
For each `VMAgent` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
name `<VMAgent-name>`.
The CRD specifies which `VMServiceScrape` should be covered by the deployed VMAgent instances based on label selection.
The Operator then generates a configuration based on the included `VMServiceScrape`s and updates the `Secret` which
contains the configuration. It continuously does so for all changes that are made to the `VMServiceScrape`s or the
`VMAgent` resource itself.
If no selection of `VMServiceScrape`s is provided - Operator leaves management of the `Secret` to the user,
so user can set custom configuration while still benefiting from the Operator's capabilities of managing VMAgent setups.
## Specification
You can see the full actual specification of the `VMAgent` resource in the **[API docs -> VMAgent](../api.md#vmagent)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## Scraping
`VMAgent` supports scraping targets with:
- [VMServiceScrape](./vmservicescrape.md),
- [VMPodScrape](./vmpodscrape.md),
- [VMNodeScrape](./vmnodescrape.md),
- [VMStaticScrape](./vmstaticscrape.md),
- [VMProbe](./vmprobe.md).
These objects tell VMAgent from which targets and how to collect metrics and
generate part of [VMAgent](./vmagent.md) scrape configuration.
For filtering scrape objects `VMAgent` uses selectors.
Selectors are defined with suffixes - `NamespaceSelector` and `Selector` for each type of scrape objects in spec of `VMAgent`:
-`serviceScrapeNamespaceSelector` and `serviceScrapeSelector` for selecting [VMServiceScrape](./vmservicescrape.md) objects,
-`podScrapeNamespaceSelector` and `podScrapeSelector` for selecting [VMPodScrape](./vmpodscrape.md) objects,
-`probeNamespaceSelector` and `probeSelector` for selecting [VMProbe](./vmprobe.md) objects,
-`staticScrapeNamespaceSelector` and `staticScrapeSelector` for selecting [VMStaticScrape](./vmstaticscrape.md) objects,
-`nodeScrapeNamespaceSelector` and `nodeScrapeSelector` for selecting [VMNodeScrape](./vmnodescrape.md) objects.
It allows configuring objects access control across namespaces and different environments.
Specification of selectors you can see in [this doc](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta).
In addition to the above selectors, the filtering of objects in a cluster is affected by the field `selectAllByDefault` of `VMAgent` spec and environment variable `WATCH_NAMESPACE` for operator.
- If `...NamespaceSelector` and `...Selector` both undefined, then by default select nothing. With option set - `spec.selectAllByDefault: true`, select all objects of given type.
- If `...NamespaceSelector` defined, `...Selector` undefined, then all objects are matching at namespaces for given `...NamespaceSelector`.
- If `...NamespaceSelector` undefined, `...Selector` defined, then all objects at `VMAgent`'s namespaces are matching for given `...Selector`.
- If `...NamespaceSelector` and `...Selector` both defined, then only objects at namespaces matched `...NamespaceSelector` for given `...Selector` are matching.
| undefined | undefined | **true** | undefined | all objects of given type (`...`) in the cluster |
| **defined** | undefined | *any* | undefined | all objects of given type (`...`) at namespaces for given `...NamespaceSelector` |
| undefined | **defined** | *any* | undefined | all objects of given type (`...`) only at `VMAgent`'s namespace are matching for given `Selector |
| **defined** | **defined** | *any* | undefined | all objects of given type (`...`) only at namespaces matched `...NamespaceSelector` for given `...Selector` |
| *any* | undefined | *any* | **defined** | all objects of given type (`...`) only at `VMAgent`'s namespace |
| *any* | **defined** | *any* | **defined** | all objects of given type (`...`) only at `VMAgent`'s namespace for given `...Selector` |
for sending data to remote storage. By default, operator set `-remoteWrite.tmpDataPath` for `VMAgent` to `/tmp` (that use k8s ephemeral storage)
and `VMAgent` loses state of the PersistentQueue on pod restarts.
In `StatefulMode``VMAgent` doesn't lose state of the PersistentQueue (file-based buffer size for unsent data) on pod restarts.
Operator creates `StatefulSet` and, with provided `PersistentVolumeClaimTemplate` at `StatefulStorage` configuration param, metrics queue is stored on disk.
**Note**: You can specify only one Secret in the VMAgent CRD configuration so use it for all additional scrape configurations.
## Relabeling
`VMAgent` supports global relabeling for all metrics and per remoteWrite target relabel config.
Note in some cases, you don't need relabeling, `key=value` label pairs can be added to the all scrapped metrics with `spec.externalLabels` for `VMAgent`:
```yaml
# simple label add config
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
externalLabels:
clusterid: some_cluster
```
`VMAgent` CR supports relabeling with [custom configMap](#relabeling-config-in-configmap)
or [inline defined at CRD](#inline-relabeling-config).
### Relabeling config in Configmap
Quick tour how to create `ConfigMap` with relabeling configuration:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: vmagent-relabel
data:
global-relabel.yaml: |
- target_label: bar
- source_labels: [aa]
separator: "foobar"
regex: "foo.+bar"
target_label: aaa
replacement: "xxx"
- action: keep
source_labels: [aaa]
- action: drop
source_labels: [aaa]
target-1-relabel.yaml: |
- action: keep_if_equal
source_labels: [foo, bar]
- action: drop_if_equal
source_labels: [foo, bar]
```
Second, add `relabelConfig` to `VMagent` spec for global relabeling with name of `Configmap` - `vmagent-relabel` and key `global-relabel.yaml`.
For relabeling per remoteWrite target, add `urlRelabelConfig` name of `Configmap` - `vmagent-relabel`
and key `target-1-relabel.yaml` to one of remoteWrite target for relabeling only for those target:
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
For using Enterprise version of [vmagent](https://docs.victoriametrics.com/vmagent.html)
you need to change version of `vmagent` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMAgent`:
After that you can pass [Kafka integration](https://docs.victoriametrics.com/vmagent.html#kafka-integration)
flags to `VMAgent` with [extraArgs](./README.md#extra-arguments).
### Reading metrics from Kafka
Here are complete example for [Reading metrics from Kafka](https://docs.victoriametrics.com/vmagent.html#reading-metrics-from-kafka):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmagent
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmagent enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: reading metrics from kafka
# more details about kafka integration you can read on https://docs.victoriametrics.com/vmagent.html#kafka-integration
# more details about these and other flags you can read on https://docs.victoriametrics.com/vmagent.html#command-line-flags-for-kafka-consumer
kafka.consumer.topic.brokers: localhost:9092
kafka.consumer.topic.format: influx
kafka.consumer.topic: metrics-by-telegraf
kafka.consumer.topic.groupID: some-id
# ...other fields...
```
### Writing metrics to Kafka
Here are complete example for [Writing metrics to Kafka](https://docs.victoriametrics.com/vmagent.html#writing-metrics-to-kafka):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmagent
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmagent enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: writing metrics to Kafka
# more details about kafka integration you can read on https://docs.victoriametrics.com/vmagent.html#kafka-integration