From cd2f0e0760426d31e66d2b82f707634c1cfe676f Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Mon, 13 Jun 2022 08:48:26 +0200 Subject: [PATCH] Readme cleanup (#2715) * docs: minor styling and wording changes Changes made after reading https://developers.google.com/tech-writing Signed-off-by: hagen1778 * docs: set proper types for code blocks Signed-off-by: hagen1778 * docs: add `copy` wrapper for some commands Signed-off-by: hagen1778 * docs: sync Signed-off-by: hagen1778 * docs: resolve conflicts Signed-off-by: hagen1778 --- README.md | 83 ++++++++++++++------- docs/README.md | 83 ++++++++++++++------- docs/Single-server-VictoriaMetrics.md | 83 ++++++++++++++------- docs/guides/k8s-monitoring-via-vm-single.md | 2 +- docs/keyConcepts.md | 4 +- 5 files changed, 168 insertions(+), 87 deletions(-) diff --git a/README.md b/README.md index 87f1a94cb..7f545f6bb 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ VictoriaMetrics has the following prominent features: * It can be used as long-term storage for Prometheus. See [these docs](#prometheus-setup) for details. * It can be used as a drop-in replacement for Prometheus in Grafana, because it supports [Prometheus querying API](#prometheus-querying-api-usage). -* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). +* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). * It features easy setup and operation: * VictoriaMetrics consists of a single [small executable](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d) without external dependencies. * All the configuration is done via explicit command-line flags with reasonable defaults. @@ -352,13 +352,17 @@ echo ' The imported data can be read via [export API](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format): +
+ ```bash curl http://localhost:8428/api/v1/export -d 'match[]=system.load.1' ``` +
+ This command should return the following output if everything is OK: -``` +```json {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]} ``` @@ -460,13 +464,17 @@ VictoriaMetrics sets the current time if the timestamp is omitted. An arbitrary number of lines delimited by `\n` (aka newline char) can be sent in one go. After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz' ``` +
+ The `/api/v1/export` endpoint should return the following response: -```bash +```json {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` @@ -745,8 +753,8 @@ More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetr ## Setting up service -Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS. -There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics). +Read [instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics +as a service for your OS. A [snap package](https://snapcraft.io/victoriametrics) is available for Ubuntu. ## How to work with snapshots @@ -839,7 +847,7 @@ for metrics to export. Use `{__name__!=""}` selector for fetching all the time s The response would contain all the data for the selected time series in [JSON streaming format](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON). Each JSON line contains samples for a single time series. An example output: -```jsonl +```json {"metric":{"__name__":"up","job":"node_exporter","instance":"localhost:9100"},"values":[0,0,0],"timestamps":[1549891472010,1549891487724,1549891503438]} {"metric":{"__name__":"up","job":"prometheus","instance":"localhost:9090"},"values":[1,1,1],"timestamps":[1549891461511,1549891476511,1549891491511]} ``` @@ -859,10 +867,14 @@ In this case the output may contain multiple lines with samples for the same tim Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: +
+ ```bash curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__!=""}' > data.jsonl.gz ``` +
+ The maximum duration for each request to `/api/v1/export` is limited by `-search.maxExportDuration` command-line flag. Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). @@ -1035,7 +1047,7 @@ curl -G 'http://localhost:8428/api/v1/export' -d 'match[]={ticker!=""}' The following response should be returned: -```bash +```json {"metric":{"__name__":"bid","market":"NASDAQ","ticker":"MSFT"},"values":[1.67],"timestamps":[1583865146520]} {"metric":{"__name__":"bid","market":"NYSE","ticker":"GOOG"},"values":[4.56],"timestamps":[1583865146495]} {"metric":{"__name__":"ask","market":"NASDAQ","ticker":"MSFT"},"values":[3.21],"timestamps":[1583865146520]} @@ -1053,29 +1065,41 @@ VictoriaMetrics accepts data in [Prometheus exposition format](https://github.co and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md) via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics: +
+ ```bash curl -d 'foo{bar="baz"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus' ``` +
+ The following command may be used for verifying the imported data: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}' ``` +
+ It should return something like the following: -``` +```json {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: +
+ ```bash # Import gzipped data to : curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz ``` +
+ Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics. @@ -1245,17 +1269,17 @@ Information about merging process is available in [single-node VictoriaMetrics]( and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176) Grafana dashboards. See more details in [monitoring docs](#monitoring). -The `merge` process is usually named "compaction", because the resulting `part` size is usually smaller than -the sum of the source `parts` because of better compression rate. The merge process provides the following additional benefits: +The `merge` process improves compression rate and keeps number of `parts` on disk relatively low. +Benefits of doing the merge process are the following: * it improves query performance, since lower number of `parts` are inspected with each query * it reduces the number of data files, since each `part` contains fixed number of files * various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling) - and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are perfomed during the merge. + and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge. Newly added `parts` either appear in the storage or fail to appear. Storage never contains partially created parts. The same applies to merge process — `parts` are either fully -merged into a new `part` or fail to merge. There are no partially merged `parts` in MergeTree. +merged into a new `part` or fail to merge. MergeTree doesn't contain partially merged `parts`. `Part` contents in MergeTree never change. Parts are immutable. They may be only deleted after the merge to a bigger `part` or when the `part` contents goes outside the configured `-retentionPeriod`. @@ -1357,9 +1381,9 @@ or similar auth proxy. ## Tuning -* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags, +* No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags, which are automatically adjusted for the available CPU and RAM resources. -* There is no need for Operating System tuning since VictoriaMetrics is optimized for default OS settings. +* No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings. The only option is increasing the limit on [the number of open files in the OS](https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a). The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk. * VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with other @@ -1376,19 +1400,23 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ## Monitoring -VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page. -These metrics may be collected by [vmagent](https://docs.victoriametrics.com/vmagent.html) -or Prometheus by adding the corresponding scrape config to it. -Alternatively they can be self-scraped by setting `-selfScrapeInterval` command-line flag to duration greater than 0. -For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page with 10 seconds interval. +VictoriaMetrics exports internal metrics in Prometheus exposition format at `/metrics` page. +These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus. +Alternatively, single-node VictoriaMetrics can self-scrape the metrics when `-selfScrapeInterval` command-line flag is +set to duration greater than 0. For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page +with 10 seconds interval. -There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176). There is also an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831). +Official Grafana dashboards available for [single-node](https://grafana.com/dashboards/10229) +and [clustered](https://grafana.com/grafana/dashboards/11176) VictoriaMetrics. +See an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831) +created by community. -Graphs on these dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. +Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it. -It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) +via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus. -The most interesting metrics are: +The most interesting health metrics are the following: * `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). @@ -1404,9 +1432,7 @@ The most interesting metrics are: If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). -VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page. - -See the example of alerting rules for VM components [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page. ## TSDB stats @@ -1791,9 +1817,10 @@ Files included in each folder: ### We kindly ask * Please don't use any other font instead of suggested. -* There should be sufficient clear space around the logo. +* To keep enough clear space around the logo. * Do not change spacing, alignment, or relative locations of the design elements. -* Do not change the proportions of any of the design elements or the design itself. You may resize as needed but must retain all proportions. +* Do not change the proportions for any of the design elements or the design itself. + You may resize as needed but must retain all proportions. ## List of command-line flags diff --git a/docs/README.md b/docs/README.md index 87f1a94cb..7f545f6bb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -33,7 +33,7 @@ VictoriaMetrics has the following prominent features: * It can be used as long-term storage for Prometheus. See [these docs](#prometheus-setup) for details. * It can be used as a drop-in replacement for Prometheus in Grafana, because it supports [Prometheus querying API](#prometheus-querying-api-usage). -* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). +* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). * It features easy setup and operation: * VictoriaMetrics consists of a single [small executable](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d) without external dependencies. * All the configuration is done via explicit command-line flags with reasonable defaults. @@ -352,13 +352,17 @@ echo ' The imported data can be read via [export API](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format): +
+ ```bash curl http://localhost:8428/api/v1/export -d 'match[]=system.load.1' ``` +
+ This command should return the following output if everything is OK: -``` +```json {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]} ``` @@ -460,13 +464,17 @@ VictoriaMetrics sets the current time if the timestamp is omitted. An arbitrary number of lines delimited by `\n` (aka newline char) can be sent in one go. After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz' ``` +
+ The `/api/v1/export` endpoint should return the following response: -```bash +```json {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` @@ -745,8 +753,8 @@ More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetr ## Setting up service -Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS. -There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics). +Read [instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics +as a service for your OS. A [snap package](https://snapcraft.io/victoriametrics) is available for Ubuntu. ## How to work with snapshots @@ -839,7 +847,7 @@ for metrics to export. Use `{__name__!=""}` selector for fetching all the time s The response would contain all the data for the selected time series in [JSON streaming format](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON). Each JSON line contains samples for a single time series. An example output: -```jsonl +```json {"metric":{"__name__":"up","job":"node_exporter","instance":"localhost:9100"},"values":[0,0,0],"timestamps":[1549891472010,1549891487724,1549891503438]} {"metric":{"__name__":"up","job":"prometheus","instance":"localhost:9090"},"values":[1,1,1],"timestamps":[1549891461511,1549891476511,1549891491511]} ``` @@ -859,10 +867,14 @@ In this case the output may contain multiple lines with samples for the same tim Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: +
+ ```bash curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__!=""}' > data.jsonl.gz ``` +
+ The maximum duration for each request to `/api/v1/export` is limited by `-search.maxExportDuration` command-line flag. Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). @@ -1035,7 +1047,7 @@ curl -G 'http://localhost:8428/api/v1/export' -d 'match[]={ticker!=""}' The following response should be returned: -```bash +```json {"metric":{"__name__":"bid","market":"NASDAQ","ticker":"MSFT"},"values":[1.67],"timestamps":[1583865146520]} {"metric":{"__name__":"bid","market":"NYSE","ticker":"GOOG"},"values":[4.56],"timestamps":[1583865146495]} {"metric":{"__name__":"ask","market":"NASDAQ","ticker":"MSFT"},"values":[3.21],"timestamps":[1583865146520]} @@ -1053,29 +1065,41 @@ VictoriaMetrics accepts data in [Prometheus exposition format](https://github.co and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md) via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics: +
+ ```bash curl -d 'foo{bar="baz"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus' ``` +
+ The following command may be used for verifying the imported data: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}' ``` +
+ It should return something like the following: -``` +```json {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: +
+ ```bash # Import gzipped data to : curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz ``` +
+ Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics. @@ -1245,17 +1269,17 @@ Information about merging process is available in [single-node VictoriaMetrics]( and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176) Grafana dashboards. See more details in [monitoring docs](#monitoring). -The `merge` process is usually named "compaction", because the resulting `part` size is usually smaller than -the sum of the source `parts` because of better compression rate. The merge process provides the following additional benefits: +The `merge` process improves compression rate and keeps number of `parts` on disk relatively low. +Benefits of doing the merge process are the following: * it improves query performance, since lower number of `parts` are inspected with each query * it reduces the number of data files, since each `part` contains fixed number of files * various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling) - and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are perfomed during the merge. + and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge. Newly added `parts` either appear in the storage or fail to appear. Storage never contains partially created parts. The same applies to merge process — `parts` are either fully -merged into a new `part` or fail to merge. There are no partially merged `parts` in MergeTree. +merged into a new `part` or fail to merge. MergeTree doesn't contain partially merged `parts`. `Part` contents in MergeTree never change. Parts are immutable. They may be only deleted after the merge to a bigger `part` or when the `part` contents goes outside the configured `-retentionPeriod`. @@ -1357,9 +1381,9 @@ or similar auth proxy. ## Tuning -* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags, +* No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags, which are automatically adjusted for the available CPU and RAM resources. -* There is no need for Operating System tuning since VictoriaMetrics is optimized for default OS settings. +* No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings. The only option is increasing the limit on [the number of open files in the OS](https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a). The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk. * VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with other @@ -1376,19 +1400,23 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ## Monitoring -VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page. -These metrics may be collected by [vmagent](https://docs.victoriametrics.com/vmagent.html) -or Prometheus by adding the corresponding scrape config to it. -Alternatively they can be self-scraped by setting `-selfScrapeInterval` command-line flag to duration greater than 0. -For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page with 10 seconds interval. +VictoriaMetrics exports internal metrics in Prometheus exposition format at `/metrics` page. +These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus. +Alternatively, single-node VictoriaMetrics can self-scrape the metrics when `-selfScrapeInterval` command-line flag is +set to duration greater than 0. For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page +with 10 seconds interval. -There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176). There is also an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831). +Official Grafana dashboards available for [single-node](https://grafana.com/dashboards/10229) +and [clustered](https://grafana.com/grafana/dashboards/11176) VictoriaMetrics. +See an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831) +created by community. -Graphs on these dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. +Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it. -It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) +via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus. -The most interesting metrics are: +The most interesting health metrics are the following: * `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). @@ -1404,9 +1432,7 @@ The most interesting metrics are: If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). -VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page. - -See the example of alerting rules for VM components [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page. ## TSDB stats @@ -1791,9 +1817,10 @@ Files included in each folder: ### We kindly ask * Please don't use any other font instead of suggested. -* There should be sufficient clear space around the logo. +* To keep enough clear space around the logo. * Do not change spacing, alignment, or relative locations of the design elements. -* Do not change the proportions of any of the design elements or the design itself. You may resize as needed but must retain all proportions. +* Do not change the proportions for any of the design elements or the design itself. + You may resize as needed but must retain all proportions. ## List of command-line flags diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index 65e3889fb..0982df819 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -37,7 +37,7 @@ VictoriaMetrics has the following prominent features: * It can be used as long-term storage for Prometheus. See [these docs](#prometheus-setup) for details. * It can be used as a drop-in replacement for Prometheus in Grafana, because it supports [Prometheus querying API](#prometheus-querying-api-usage). -* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). +* It can be used as a drop-in replacement for Graphite in Grafana, because it supports [Graphite API](#graphite-api-usage). * It features easy setup and operation: * VictoriaMetrics consists of a single [small executable](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d) without external dependencies. * All the configuration is done via explicit command-line flags with reasonable defaults. @@ -356,13 +356,17 @@ echo ' The imported data can be read via [export API](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format): +
+ ```bash curl http://localhost:8428/api/v1/export -d 'match[]=system.load.1' ``` +
+ This command should return the following output if everything is OK: -``` +```json {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]} ``` @@ -464,13 +468,17 @@ VictoriaMetrics sets the current time if the timestamp is omitted. An arbitrary number of lines delimited by `\n` (aka newline char) can be sent in one go. After that the data may be read via [/api/v1/export](#how-to-export-data-in-json-line-format) endpoint: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz' ``` +
+ The `/api/v1/export` endpoint should return the following response: -```bash +```json {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` @@ -749,8 +757,8 @@ More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetr ## Setting up service -Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS. -There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics). +Read [instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics +as a service for your OS. A [snap package](https://snapcraft.io/victoriametrics) is available for Ubuntu. ## How to work with snapshots @@ -843,7 +851,7 @@ for metrics to export. Use `{__name__!=""}` selector for fetching all the time s The response would contain all the data for the selected time series in [JSON streaming format](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON). Each JSON line contains samples for a single time series. An example output: -```jsonl +```json {"metric":{"__name__":"up","job":"node_exporter","instance":"localhost:9100"},"values":[0,0,0],"timestamps":[1549891472010,1549891487724,1549891503438]} {"metric":{"__name__":"up","job":"prometheus","instance":"localhost:9090"},"values":[1,1,1],"timestamps":[1549891461511,1549891476511,1549891491511]} ``` @@ -863,10 +871,14 @@ In this case the output may contain multiple lines with samples for the same tim Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data: +
+ ```bash curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__!=""}' > data.jsonl.gz ``` +
+ The maximum duration for each request to `/api/v1/export` is limited by `-search.maxExportDuration` command-line flag. Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format). @@ -1039,7 +1051,7 @@ curl -G 'http://localhost:8428/api/v1/export' -d 'match[]={ticker!=""}' The following response should be returned: -```bash +```json {"metric":{"__name__":"bid","market":"NASDAQ","ticker":"MSFT"},"values":[1.67],"timestamps":[1583865146520]} {"metric":{"__name__":"bid","market":"NYSE","ticker":"GOOG"},"values":[4.56],"timestamps":[1583865146495]} {"metric":{"__name__":"ask","market":"NASDAQ","ticker":"MSFT"},"values":[3.21],"timestamps":[1583865146520]} @@ -1057,29 +1069,41 @@ VictoriaMetrics accepts data in [Prometheus exposition format](https://github.co and in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md) via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics: +
+ ```bash curl -d 'foo{bar="baz"} 123' -X POST 'http://localhost:8428/api/v1/import/prometheus' ``` +
+ The following command may be used for verifying the imported data: +
+ ```bash curl -G 'http://localhost:8428/api/v1/export' -d 'match={__name__=~"foo"}' ``` +
+ It should return something like the following: -``` +```json {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]} ``` Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data: +
+ ```bash # Import gzipped data to : curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz ``` +
+ Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args. For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics. @@ -1249,17 +1273,17 @@ Information about merging process is available in [single-node VictoriaMetrics]( and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176) Grafana dashboards. See more details in [monitoring docs](#monitoring). -The `merge` process is usually named "compaction", because the resulting `part` size is usually smaller than -the sum of the source `parts` because of better compression rate. The merge process provides the following additional benefits: +The `merge` process improves compression rate and keeps number of `parts` on disk relatively low. +Benefits of doing the merge process are the following: * it improves query performance, since lower number of `parts` are inspected with each query * it reduces the number of data files, since each `part` contains fixed number of files * various background maintenance tasks such as [de-duplication](#deduplication), [downsampling](#downsampling) - and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are perfomed during the merge. + and [freeing up disk space for the deleted time series](#how-to-delete-time-series) are performed during the merge. Newly added `parts` either appear in the storage or fail to appear. Storage never contains partially created parts. The same applies to merge process — `parts` are either fully -merged into a new `part` or fail to merge. There are no partially merged `parts` in MergeTree. +merged into a new `part` or fail to merge. MergeTree doesn't contain partially merged `parts`. `Part` contents in MergeTree never change. Parts are immutable. They may be only deleted after the merge to a bigger `part` or when the `part` contents goes outside the configured `-retentionPeriod`. @@ -1361,9 +1385,9 @@ or similar auth proxy. ## Tuning -* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags, +* No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags, which are automatically adjusted for the available CPU and RAM resources. -* There is no need for Operating System tuning since VictoriaMetrics is optimized for default OS settings. +* No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings. The only option is increasing the limit on [the number of open files in the OS](https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a). The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk. * VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with other @@ -1380,19 +1404,23 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ## Monitoring -VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page. -These metrics may be collected by [vmagent](https://docs.victoriametrics.com/vmagent.html) -or Prometheus by adding the corresponding scrape config to it. -Alternatively they can be self-scraped by setting `-selfScrapeInterval` command-line flag to duration greater than 0. -For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page with 10 seconds interval. +VictoriaMetrics exports internal metrics in Prometheus exposition format at `/metrics` page. +These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus. +Alternatively, single-node VictoriaMetrics can self-scrape the metrics when `-selfScrapeInterval` command-line flag is +set to duration greater than 0. For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page +with 10 seconds interval. -There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176). There is also an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831). +Official Grafana dashboards available for [single-node](https://grafana.com/dashboards/10229) +and [clustered](https://grafana.com/grafana/dashboards/11176) VictoriaMetrics. +See an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831) +created by community. -Graphs on these dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it. +Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it. -It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) +via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus. -The most interesting metrics are: +The most interesting health metrics are the following: * `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). @@ -1408,9 +1436,7 @@ The most interesting metrics are: If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series). -VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page. - -See the example of alerting rules for VM components [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml). +VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page. ## TSDB stats @@ -1795,9 +1821,10 @@ Files included in each folder: ### We kindly ask * Please don't use any other font instead of suggested. -* There should be sufficient clear space around the logo. +* To keep enough clear space around the logo. * Do not change spacing, alignment, or relative locations of the design elements. -* Do not change the proportions of any of the design elements or the design itself. You may resize as needed but must retain all proportions. +* Do not change the proportions for any of the design elements or the design itself. + You may resize as needed but must retain all proportions. ## List of command-line flags diff --git a/docs/guides/k8s-monitoring-via-vm-single.md b/docs/guides/k8s-monitoring-via-vm-single.md index 2ae3832b8..973df57ee 100644 --- a/docs/guides/k8s-monitoring-via-vm-single.md +++ b/docs/guides/k8s-monitoring-via-vm-single.md @@ -74,7 +74,7 @@ Run this command in your terminal:
.html -```yaml +```bash helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml ``` diff --git a/docs/keyConcepts.md b/docs/keyConcepts.md index 61746e597..11ffef800 100644 --- a/docs/keyConcepts.md +++ b/docs/keyConcepts.md @@ -145,13 +145,13 @@ vm_per_query_rows_processed_count_count 11 In practice, histogram `vm_per_query_rows_processed_count` may be used in the following way: -```Go +```go // define the histogram perQueryRowsProcessed := metrics.NewHistogram(`vm_per_query_rows_processed_count`) // use the histogram during processing for _, query := range queries { -perQueryRowsProcessed.Update(len(query.Rows)) + perQueryRowsProcessed.Update(len(query.Rows)) } ```