From 46e6fdb131e43816a58741f91cb1ef5ea2057fe6 Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Sat, 10 Apr 2021 13:46:08 +0100 Subject: [PATCH] Docs update (#1199) * docs: drop table of contents for `vmctl` We already have it autogenerated on .github.io, so no need to keep it. * docs: mention OpenTSDB migration feature for vmctl * docs: sync docs for `vmalert` --- app/vmctl/README.md | 30 ------------------------------ docs/CHANGELOG.md | 2 ++ docs/vmalert.md | 5 +++++ docs/vmctl.md | 28 ++++++++++++++++++++++++++++ 4 files changed, 35 insertions(+), 30 deletions(-) diff --git a/app/vmctl/README.md b/app/vmctl/README.md index f9cc635546..166171a65d 100644 --- a/app/vmctl/README.md +++ b/app/vmctl/README.md @@ -10,36 +10,6 @@ Features: - [x] OpenTSDB: migrate data from OpenTSDB to VictoriaMetrics - [ ] Storage Management: data re-balancing between nodes -# Table of contents - -* [Articles](#articles) -* [How to build](#how-to-build) -* [Migrating data from OpenTSDB](#migrating-data-from-opentsdb) - * [Retention Strings](#retention-strings) - * [Restarting OpenTSDB Migrations](#restarting-opentsdb-migrations) -* [Migrating data from InfluxDB 1.x](#migrating-data-from-influxdb-1x) - * [Data mapping](#data-mapping) - * [Configuration](#configuration) - * [Filtering](#filtering) -* [Migrating data from InfluxDB 2.x](#migrating-data-from-influxdb-2x) -* [Migrating data from Prometheus](#migrating-data-from-prometheus) - * [Data mapping](#data-mapping-1) - * [Configuration](#configuration-1) - * [Filtering](#filtering-1) -* [Migrating data from Thanos](#migrating-data-from-thanos) - * [Current data](#current-data) - * [Historical data](#historical-data) -* [Migrating data from VictoriaMetrics](#migrating-data-from-victoriametrics) - * [Native protocol](#native-protocol) -* [Tuning](#tuning) - * [Influx mode](#influx-mode) - * [Prometheus mode](#prometheus-mode) - * [VictoriaMetrics importer](#victoriametrics-importer) - * [Importer stats](#importer-stats) -* [Significant figures](#significant-figures) -* [Adding extra labels](#adding-extra-labels) - - ## Articles * [How to migrate data from Prometheus](https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 60c2371b7a..9f9d4a0723 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -6,6 +6,8 @@ sort: 13 ## tip +* FEATURE: add OpenTSDB migration option to vmctl. See more details [here](https://victoriametrics.github.io/vmctl#migrating-data-from-opentsdb). +Thanks to @johnseekins! ## [v1.58.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.58.0) diff --git a/docs/vmalert.md b/docs/vmalert.md index c1e912aa71..5f9bcc9604 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -135,6 +135,11 @@ annotations: [ : ] ``` +It is allowed to use [Go templating](https://golang.org/pkg/text/template/) in annotations +to format data, iterate over it or execute expressions. +Additionally, `vmalert` provides some extra templating functions +listed [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/notifier/template_func.go). + #### Recording rules The syntax for recording rules is following: diff --git a/docs/vmctl.md b/docs/vmctl.md index 5fed39e363..4992bfe798 100644 --- a/docs/vmctl.md +++ b/docs/vmctl.md @@ -11,6 +11,7 @@ Features: - [x] Thanos: migrate data from Thanos to VictoriaMetrics - [ ] ~~Prometheus: migrate data from Prometheus to VictoriaMetrics by query~~(discarded) - [x] InfluxDB: migrate data from InfluxDB to VictoriaMetrics +- [x] OpenTSDB: migrate data from OpenTSDB to VictoriaMetrics - [ ] Storage Management: data re-balancing between nodes ## Articles @@ -64,6 +65,33 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b 2. Run `make vmctl-arm-prod` or `make vmctl-arm64-prod` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics). It builds `vmctl-arm-prod` or `vmctl-arm64-prod` binary respectively and puts it into the `bin` folder. +## Migrating data from OpenTSDB + +`vmctl` supports the `opentsdb` mode to migrate data from OpenTSDB to VictoriaMetrics time-series database. + +See `./vmctl opentsdb --help` for details and full list of flags. + +*OpenTSDB migration is not possible without a functioning [meta](http://opentsdb.net/docs/build/html/user_guide/metadata.html) table to search for metrics/series.* + +OpenTSDB migration works like so: + +1. Find metrics based on selected filters (or the default filter set ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']) + * e.g. `curl -Ss "http://opentsdb:4242/api/suggest?type=metrics&q=sys"` +2. Find series associated with each returned metric + * e.g. `curl -Ss "http://opentsdb:4242/api/search/lookup?m=system.load5&limit=1000000"` +3. Download data for each series in chunks defined in the CLI switches + * e.g. `-retention=sum-1m-avg:1h:90d` == + * `curl -Ss "http://opentsdb:4242/api/query?start=1h-ago&end=now&m=sum:1m-avg-none:system.load5\{host=host1\}"` + * `curl -Ss "http://opentsdb:4242/api/query?start=2h-ago&end=1h-ago&m=sum:1m-avg-none:system.load5\{host=host1\}"` + * `curl -Ss "http://opentsdb:4242/api/query?start=3h-ago&end=2h-ago&m=sum:1m-avg-none:system.load5\{host=host1\}"` + * ... + * `curl -Ss "http://opentsdb:4242/api/query?start=2160h-ago&end=2159h-ago&m=sum:1m-avg-none:system.load5\{host=host1\}"` + +This means that we must stream data from OpenTSDB to VictoriaMetrics in chunks. This is where concurrency for OpenTSDB comes in. We can query multiple chunks at once, but we shouldn't perform too many chunks at a time to avoid overloading the OpenTSDB cluster. + +### Restarting OpenTSDB migrations + +One important note for OpenTSDB migration: Queries/HBase scans can "get stuck" within OpenTSDB itself. This can cause instability and performance issues within an OpenTSDB cluster, so stopping the migrator to deal with it may be necessary. Because of this, we provide the timstamp we started collecting data from at thebeginning of the run. You can stop and restart the importer using this "hard timestamp" to ensure you collect data from the same time range over multiple runs. ## Migrating data from InfluxDB (1.x)