VictoriaMetrics/deployment/docker
Zakhar Bessarab 5b3cbd4db1
app/vlinsert: add support of loki push protocol (#4482)
* app/vlinsert: add support of loki push protocol

- implemented loki push protocol for both Protobuf and JSON formats
- added examples in documentation
- added example docker-compose

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert: move protobuf metric into its own file

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: update reference to docker image

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: make volume name unique

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: add license reference

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: fix volume name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: move entities to places where those are used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: refactor to use common components

- use CommonParameters from insertutils
- stop ingestion after first error similar to elasticsearch and jsonline

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: address review feedback

- add missing logstorage.PutLogRows calls
- refactor tenant ID parsing to use common function
- reduce number of allocations for parsing by reusing  logfields slices
- add tests and benchmarks for requests processing funcs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 16:49:43 -07:00
..
base all: run apk update && apk upgrade in base Alpine Docker image in order to get all the recent security fixes 2023-02-09 14:03:02 -08:00
builder CI: speedup build by 2.4x. restore nightly build (#3772) 2023-02-07 09:31:00 -08:00
provisioning deployment/docker/provisioning/dashboards: sync with master branch after 5714a68ac6 2022-09-21 12:06:01 +03:00
victorialogs app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 16:49:43 -07:00
alertmanager.yml deployment: add vmalert+alertmanager services and list of default alerts for cluster version (#1187) 2021-04-05 22:29:04 +03:00
alerts-cluster.yml docs/Troubleshooting.md: document an additional case, which could result in slow inserts 2023-03-20 14:33:27 -07:00
alerts-health.yml add vmalertmanager filter for health alerts (#4665) 2023-07-19 14:50:06 -07:00
alerts-vmagent.yml update links to grafana dashboards (#3534) 2022-12-28 11:22:02 -08:00
alerts-vmalert.yml alerts: decrease severity to info for RecordingRulesNoData (#4089) 2023-05-08 15:46:15 -07:00
alerts.yml docs/Troubleshooting.md: document an additional case, which could result in slow inserts 2023-03-20 14:33:27 -07:00
docker-compose-cluster.yml bump to 1.91.3 (#4565) 2023-07-06 22:10:54 -07:00
docker-compose.yml bump to 1.91.3 (#4565) 2023-07-06 22:10:54 -07:00
Makefile all: update Go builder from 1.20.5 to 1.20.6 2023-07-12 01:00:24 -07:00
prometheus-cluster.yml deployment/docker: move cluster compose env to master branch (#3130) 2022-09-21 12:03:10 +03:00
prometheus.yml deployment/docker: sync with master branch 2023-01-23 22:59:42 -08:00
README.md docs: fix links to alerts and alertmanager (#3829) 2023-02-18 15:19:48 -08:00

Docker compose environment for VictoriaMetrics

Docker compose environment for VictoriaMetrics includes VictoriaMetrics components, Alertmanager and Grafana.

For starting the docker-compose environment ensure you have docker installed and running and access to the Internet. All commands should be executed from the root directory of the repo.

To spin-up environment for single server VictoriaMetrics run the following command:

make docker-single-up

To shut down the docker-compose environment for single server run the following command:

make docker-single-down

For cluster version the command will be the following:

make docker-cluster-up

To shut down the docker compose environment for cluster version run the following command:

make docker-cluster-down

VictoriaMetrics single server

VictoriaMetrics will be accessible on the following ports:

  • --graphiteListenAddr=:2003
  • --opentsdbListenAddr=:4242
  • --httpListenAddr=:8428

The communication scheme between components is the following:

  • vmagent sends scraped metrics to single server VictoriaMetrics;
  • grafana is configured with datasource pointing to single server VictoriaMetrics;
  • vmalert is configured to query single server VictoriaMetrics and send alerts state and recording rules back to it;
  • alertmanager is configured to receive notifications from vmalert.

To access vmalert use link http://localhost:8428/vmalert.

To access vmui use link http://localhost:8428/vmui.

VictoriaMetrics cluster

VictoriaMetrics cluster environment consists of vminsert, vmstorage and vmselect components. vmselect has exposed port :8481, vminsert has exposed port :8480 and the rest of components are available only inside the environment.

The communication scheme between components is the following:

  • vmagent sends scraped metrics to vminsert;
  • vminsert forwards data to vmstorage;
  • vmselect is connected to vmstorage for querying data;
  • grafana is configured with datasource pointing to vmselect;
  • vmalert is configured to query vmselect and send alerts state and recording rules to vminsert;
  • alertmanager is configured to receive notifications from vmalert.

To access vmalert use link http://localhost:8481/select/0/prometheus/vmalert.

To access vmui use link http://localhost:8481/select/0/prometheus/vmui.

vmagent

vmagent is used for scraping and pushing time series to VictoriaMetrics instance. It accepts Prometheus-compatible configuration prometheus.yml with listed targets for scraping.

Web interface link.

vmalert

vmalert evaluates alerting rules alerts.yml to track VictoriaMetrics health state. It is connected with AlertManager for firing alerts, and with VictoriaMetrics for executing queries and storing alert's state.

Web interface link.

alertmanager

AlertManager accepts notifications from vmalert and fires alerts. All notifications are blackholed according to alertmanager.yml config.

Web interface link.

Grafana

To access service open following link.

Default credential:

  • login - admin
  • password - admin

Grafana is provisioned by default with following entities:

  • VictoriaMetrics datasource
  • VictoriaMetrics - cluster datasource
  • VictoriaMetrics overview dashboard
  • VictoriaMetrics - cluster dashboard
  • VictoriaMetrics - vmagent dashboard
  • VictoriaMetrics - vmalert dashboard

Remember to pick VictoriaMetrics - cluster datasource when viewing VictoriaMetrics - cluster dashboard.