diff --git a/docs/data-ingestion/Proxmox.md b/docs/data-ingestion/Proxmox.md new file mode 100644 index 0000000000..c1ae597607 --- /dev/null +++ b/docs/data-ingestion/Proxmox.md @@ -0,0 +1,71 @@ +--- +title: Proxmox +weight: 1 +sort: 1 +menu: + docs: + identifier: "proxmox" + parent: "data-ingestion" + weight: 1 + # sort: 1 +aliases: + - /data-ingestion/proxmox.html + - /data-ingestion/Proxmox.html +--- + +# Proxmox Data Ingestion +Since Proxmox Virtual Environment(PVE) and Proxmox Backup Server(PBS) support sending data using the InfluxDB We can use the InfluxDB write support built into VictoriaMetrics +Currently PVE and PBS only support using an Authorization Token for authentication and does not support basic auth or a username and password. + +## Proxmox Virtual Environment (PVE) +If want help Sending your data to Managed VictoriaMetrics check out [our blog](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/). + +1. Login to PVE as an administrator +2. Go to DataCenter > MetricServer > Add > InfluxDB + +![PVE Metric Navigation](pve-nav.webp) + + +3. Set the parameters as follows: + - Name: VictoriaMetrics (can be changed to any string) + - Server: the hostname or IP of your VictoriaMetrics Instance + - Port: This will vary depending how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html) + - Protocol: use HTTPS if you have TLS/SSL configured otherwise use HTTP + - Organization: leave empty since it doesn't get used + - Bucket: leave empty since it doesn't get used + - Token: your token from vmauth or leave blank if you don't have authentication enabled + - If you need to ignore TLS/SSL errors check the advanced box and uncheck the verify certificate box +4. Click the `Create` button + +![PVE Metric Form](pve-form.webp) + +5. Run `system_uptime{object="nodes"}` in vmui or in the explore view in Grafana to verify metrics from PVE are being sent to VictoriaMetrics. +You should see 1 time series per node in your PVE cluster. + +## Proxmox Backup Server (PBS) +1. Login to PBS as an administrator +2. Go to Configuration > Metrics Server > Add > InfluxDB + + +![PBS Metric Navigation](pbs-nav.webp) + + +3. Set the parameters as follows: + - Name: VictoriaMetrics (can be set to any string) + - URL: http(s)://: + - set the URL to https if you have TLS enabled and http if you do not + - Port: This will vary depending how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html) + - Organization: leave empty since it doesn't get used + - Bucket: leave empty since it doesn't get used + - Token: your token from vmauth or leave blank if you don't have authentication enabled +4. Click the `Create` button + + +![PBS Metric Form](pbs-form.webp) + + +5. Run `cpustat_idle{object="host"}` in vmui or in the explore view in Grafana to verify metrics from PBS are being to VictoriaMetrics. + + +# References +- [Blog Post for configuring Managed VictoriaMetrics and Proxmox VE](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/) diff --git a/docs/data-ingestion/README.md b/docs/data-ingestion/README.md new file mode 100644 index 0000000000..ae16413d77 --- /dev/null +++ b/docs/data-ingestion/README.md @@ -0,0 +1,35 @@ +--- +# sort: 14 +title: Data Ingestion +weight: 0 +menu: + docs: + parent: 'victoriametrics' + identifier: 'data-ingestion' + weight: 7 +aliases: +- /data-ingestion.html +- /data-ingestion.html +- /dataingestion/ +--- + +# Data Ingestion +In This Folder you will find instructions for sending data to VictoriaMetrics from a variety of platforms. +If your tool is not listed it is likely you can ingest your data into VictoriaMetrics using one of the protocols listed in our [Prominent features]({{< ref "/Single-server-VictoriaMetrics.md#prominent-features" >}}) section. + +If you are unsure what port number to use when pushing data to VictoriaMetrics single node, vminsert, vmagent, and vmauth we have listed the default ports below. + +- VictoriaMetrics Single: 8428 +- vmagent: 8429 +- vmauth: 8427 +- vminsert: 8482 + +In the rest of the documentation we will assume you have configured your push endpoint to use TLS/SSL on port 443 so the urls in the rest of the documentation will look like `https://` instead of `http://:8428` for VictoriaMetrics single. + +## Documented Collectors/Agents +* [Telegraf]({{< relref "Telegraf.md" >}}) +* [Vector]({{< relref "Vector.md" >}}) + +## Supported Platforms +* [Proxmox Virtual Environment and Proxmox Backup Server]({{< relref "Proxmox.md" >}}) + diff --git a/docs/data-ingestion/Telegraf.md b/docs/data-ingestion/Telegraf.md new file mode 100644 index 0000000000..744dcd1cd0 --- /dev/null +++ b/docs/data-ingestion/Telegraf.md @@ -0,0 +1,105 @@ +--- +title: Telegraf +weight: 1 +sort: 1 +menu: + docs: + identifier: "telegraf" + parent: "data-ingestion" + weight: 1 + # sort: 1 +aliases: + - /data-ingestion/telegraf.html + - /data-ingestion/Telegraf.html +--- +# Telegraf Setup +You will need to add the following output section to a Telegraf configuration file and reload Telegraf to enable shipping data from Telegraf to VictoriaMetrics. +All the options examples below can be combined to fit your use case + +To avoid storing Passwords in configuration files you can store as a key value pair in `/etc/default/telegraf` on Linux as follows +``` +victoriametrics_url="https://metrics.example.com" +victoriametrics_user="telegraf" +victoriametrics_password="password" +victoriametrics_token="my_token" +``` +and they can be referenced in a Telegraf configuration file by prepending the variable name with `$` ex. `$victoriametrics_url` will be translated to `https://metrics.example.com` if it referenced in a Telegraf configuration using the values from `/etc/default/telegraf` in the values seen above. +Otherwise please replace the variables below to fit your setup. + +If you want to mimic this behavior on windows please read [Influx Data's blog on storing variables in the registry](https://www.influxdata.com/blog/using-telegraf-on-windows/) + +## Minimum Configuration with no Authentication +```toml +[[outputs.influxdb]] + urls = ["$victoriametrics_url"] + database = "victoriametrics" + skip_database_creation = true + exclude_retention_policy_tag = true + content_encoding = "gzip" +``` + + +## HTTP Basic Authentication (Username and Password) +This is the same as the minimum configuration, but adds the `username` and `password` options + +```toml +[[outputs.influxdb]] + urls = ["$victoriametrics_url"] + username = "$victoriametrics_user" + password = "$victoriametrics_password" + database = "victoriametrics" + skip_database_creation = true + exclude_retention_policy_tag = true + content_encoding = "gzip" +``` + +## Bearer Authentication (Token) + +This is the same as the minimum configuration but adds the authorization header + +``` +[[outputs.influxdb]] + urls = ["$victoriametrics_url"] + http_headers = {"Authorization" = "Bearer $victoriametrics_token"} + database = "victoriametrics" + skip_database_creation = true + exclude_retention_policy_tag = true + content_encoding = "gzip" +``` + +## Route certain metrics +If you only want to route certain metrics to VictoriaMetrics use the `namepass` option with a comma separated listed of the measurements you wish to send to VictoriaMetrics. + +``` +[[outputs.influxdb]] + urls = ["$victoriametrics_url"] + username = "$victoriametrics_user" + password = "$victoriametrics_password" + database = "victoriametrics" + skip_database_creation = true + exclude_retention_policy_tag = true + content_encoding = "gzip" + namepass = ["cpu","disk","measurement1","measurement2"] +``` + +## Ignore TLS/SSL Certificate errors +This is the same as the minimum configuration but adds `insecure_skip_verify = true` to the configuration to ignore TLS certificate errors. +This is not recommended since it can allow sending metrics to a compromised site. + +``` +[[outputs.influxdb]] + urls = ["$victoriametrics_url"] + username = "$victoriametrics_user" + password = "$victoriametrics_password" + database = "victoriametrics" + skip_database_creation = true + exclude_retention_policy_tag = true + content_encoding = "gzip" + insecure_skip_verify = true +``` + +# References +- [Install Telegraf](https://docs.influxdata.com/telegraf/v1/install/) +- [InfluxDBv1 output for Telegraf](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/influxdb) +- [Storing Telegraf variables in the windows registry](https://www.influxdata.com/blog/using-telegraf-on-windows/) +- [Telegraf variables](https://docs.influxdata.com/telegraf/v1/configuration/#example-telegraf-environment-variables) diff --git a/docs/data-ingestion/Vector.md b/docs/data-ingestion/Vector.md new file mode 100644 index 0000000000..482d851ec0 --- /dev/null +++ b/docs/data-ingestion/Vector.md @@ -0,0 +1,121 @@ +--- +title: Vector +weight: 1 +sort: 1 +menu: + docs: + identifier: "Vector" + parent: "data-ingestion" + weight: 1 + # sort: 1 +aliases: + - /data-ingestion/Vector.html + - /data-ingestion/vector.html +--- +# Vector +To Send data to Vector you need to configure with a Prometheus remote write sink and forward metrics to that sink from at least 1 source. +You will need to replace the values in `<>` with your to match your setup. + +## Minimum Config +```yaml +sources: + host_metrics_source: + type: host_metrics +sinks: + victoriametrics_sink: + type: prometheus_remote_write + inputs: + - host_metrics_source + endpoint: "https:///api/v1/write" + healthcheck: + enabled: false +``` + +## Basic Authentication + +This adds support for basic authentication by defining the auth strategy, user, and password fields: + + +```yaml +sources: + host_metrics_source: + type: host_metrics +sinks: + victoriametrics_sink: + type: prometheus_remote_write + inputs: + - host_metrics_source + endpoint: "https:///api/v1/write" + auth: + strategy: "basic" + user: "" + healthcheck: + enabled: false + +``` + +## Bearer / Token Authentication + +This adds support for bearer/token authentication by defining the auth strategy and token fields: + + +```yaml +sources: + host_metrics_source: + type: host_metrics +sinks: + victoriametrics_sink: + type: prometheus_remote_write + inputs: + - host_metrics_source + endpoint: "https:///api/v1/write" + auth: + strategy: "bearer" + token: "" + healthcheck: + enabled: false +``` + +## VictoriaMetrics and VictoriaLogs + +This combines the Bearer Authentication section with the [VictoriaLogs docs for Vector](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/), +so you can send metrics and logs with 1 agent to multiple sources: + + +```yaml +sources: + host_metrics_source: + type: host_metrics + journald_source: + type: journald +sinks: + victoriametrics_sink: + type: prometheus_remote_write + inputs: + - host_metrics_source + endpoint: "https:///api/v1/write" + auth: + strategy: "bearer" + token: "" + healthcheck: + enabled: false + victorialogs_sink: + inputs: + - journald_source + type: elasticsearch + endpoints: + - "https:///insert/elasticsearch/" + mode: bulk + api_version: "v8" + healthcheck: + enabled: false + query: + _msg_field: "message" + _time_field: "timestamp" + _stream_fields: "host,container_name" +``` + +# References +- [Vector documentation](https://vector.dev/docs/) +- [VictoriaLogs documenation for using vector]({{< ref "/victorialogs/data-ingestion/vector" >}}) diff --git a/docs/data-ingestion/pbs-form.webp b/docs/data-ingestion/pbs-form.webp new file mode 100644 index 0000000000..7a16bb2042 Binary files /dev/null and b/docs/data-ingestion/pbs-form.webp differ diff --git a/docs/data-ingestion/pbs-nav.webp b/docs/data-ingestion/pbs-nav.webp new file mode 100644 index 0000000000..a775638219 Binary files /dev/null and b/docs/data-ingestion/pbs-nav.webp differ diff --git a/docs/data-ingestion/pve-form.webp b/docs/data-ingestion/pve-form.webp new file mode 100644 index 0000000000..7293c5b157 Binary files /dev/null and b/docs/data-ingestion/pve-form.webp differ diff --git a/docs/data-ingestion/pve-nav.webp b/docs/data-ingestion/pve-nav.webp new file mode 100644 index 0000000000..ed5d9a13e5 Binary files /dev/null and b/docs/data-ingestion/pve-nav.webp differ