diff --git a/CHANGELOG.md b/CHANGELOG.md index ddf86a5..b08e3f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,7 +30,7 @@ The following tip changes can be tested by building VictoriaMetrics components f * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442). -* FEATURE: vmstorage: suppress "broken pipe" errors for search queries on vmstorage side. See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418/commits/a6a7795b9e1f210d614a2c5f9a3016b97ded4792). +* FEATURE: vmstorage: suppress "broken pipe" and "connection reset by peer" errors for search queries on vmstorage side. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418/commits/a6a7795b9e1f210d614a2c5f9a3016b97ded4792) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4498/commits/830dac177f0f09032165c248943a5da0e10dfe90) commits. * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add panel for tracking rate of syscalls while writing or reading from disk via `process_io_(read|write)_syscalls_total` metrics. * FEATURE: accept timestamps in milliseconds at `start`, `end` and `time` query args in [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage). See [these docs](https://docs.victoriametrics.com/#timestamp-formats) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459). diff --git a/VictoriaLogs/QuickStart.md b/VictoriaLogs/QuickStart.md index 57a093b..911acbb 100644 --- a/VictoriaLogs/QuickStart.md +++ b/VictoriaLogs/QuickStart.md @@ -8,9 +8,37 @@ before you start working with VictoriaLogs. There are the following options exist: +- [To run pre-built binaries](#pre-built-binaries) - [To run Docker image](#docker-image) +- [To run in Kubernetes with Helm charts](#helm-charts) - [To build VictoriaLogs from source code](#building-from-source-code) +### Pre-built binaries + +Pre-built binaries for VictoriaLogs are availble at the [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/) page. +Just download archive for the needed Operating system and architecture, unpack it and run `victoria-logs-prod` from it. + +For example, the following commands download VictoriaLogs archive for Linux/amd64, unpack and run it: + +```bash +curl https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v0.1.0-victorialogs/victoria-logs-linux-amd64-v0.1.0-victorialogs.tar.gz +tar xzf victoria-logs-linux-amd64-v0.1.0-victorialogs.tar.gz +./victoria-logs-prod +``` + +VictoriaLogs is ready for [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +and [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) at the TCP port `9428` now! +It has no any external dependencies, so it may run in various environments without additional setup and configuration. +VictoriaLogs automatically adapts to the available CPU and RAM resources. It also automatically setups and creates +the needed indexes during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). + +See also: + +- [How to configure VictoriaLogs](#how-to-configure-victorialogs) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) + + ### Docker image You can run VictoriaLogs in a Docker container. It is the easiest way to start using VictoriaLogs. @@ -18,9 +46,20 @@ Here is the command to run VictoriaLogs in a Docker container: ```bash docker run --rm -it -p 9428:9428 -v ./victoria-logs-data:/victoria-logs-data \ - docker.io/victoriametrics/victoria-logs:heads-public-single-node-0-ga638f5e2b + docker.io/victoriametrics/victoria-logs:v0.1.0-victorialogs ``` +See also: + +- [How to configure VictoriaLogs](#how-to-configure-victorialogs) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) + +### Helm charts + +You can run VictoriaLogs in Kubernetes environment +with [these Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-logs-single/README.md). + ### Building from source code Follow the following steps in order to build VictoriaLogs from source code: @@ -50,11 +89,23 @@ It has no any external dependencies, so it may run in various environments witho VictoriaLogs automatically adapts to the available CPU and RAM resources. It also automatically setups and creates the needed indexes during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). -It is possible to change the TCP port via `-httpListenAddr` command-line flag. For example, the following command -starts VictoriaLogs, which accepts incoming requests at port `9200` (aka ElasticSearch HTTP API port): +See also: + +- [How to configure VictoriaLogs](#how-to-configure-victorialogs) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) + + +## How to configure VictoriaLogs + +VictoriaLogs is configured via command-line flags. All the command-line flags have sane defaults, +so there is no need in tuning them in general case. VictoriaLogs runs smoothly in most environments +without additional configuration. + +Pass `-help` to VictoriaLogs in order to see the list of supported command-line flags with their description and default values: ```bash -/path/to/victoria-logs -httpListenAddr=:9200 +/path/to/victoria-logs -help ``` VictoriaLogs stores the ingested data to the `victoria-logs-data` directory by default. The directory can be changed @@ -66,3 +117,22 @@ E.g. it uses the retention of 7 days. Read [these docs](https://docs.victoriamet for the [ingested](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) logs. It is recommended setting up monitoring of VictoriaLogs according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/#monitoring). + +See also: + +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) + +## Docker demos + +Here are a Docker-compose demos, which start VictoriaLogs and push logs to it via various log collectors: + +- [Filebeat demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/filebeat-docker) +- [Fluentbit demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker) +- [Logstash demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash) +- [Vector demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-docker) + +You can use [this Helm chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-logs-single/README.md) +as a demo for running Fluentbit in Kubernetes with VictoriaLogs. + + diff --git a/VictoriaLogs/README.md b/VictoriaLogs/README.md index bb55e0c..230ea70 100644 --- a/VictoriaLogs/README.md +++ b/VictoriaLogs/README.md @@ -12,11 +12,13 @@ It provides the following key features: see [LogsQL docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html). - VictoriaLogs can be seamlessly combined with good old Unix tools for log analysis such as `grep`, `less`, `sort`, `jq`, etc. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line) for details. -- VictoriaLogs capacity and performance scales lineraly with the available resources (CPU, RAM, disk IO, disk space). +- VictoriaLogs capacity and performance scales linearly with the available resources (CPU, RAM, disk IO, disk space). It runs smoothly on both Raspberry PI and a server with hundreds of CPU cores and terabytes of RAM. - VictoriaLogs can handle much bigger data volumes than ElasticSearch and Grafana Loki when running on comparable hardware. + See [these docs](#benchmarks). - VictoriaLogs supports multitenancy - see [these docs](#multitenancy). - VictoriaLogs supports out of order logs' ingestion aka backfilling. +- VictoriaLogs provides simple web UI for querying logs - see [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#web-ui). VictoriaLogs is at Preview stage now. It is ready for evaluation in production and verifying claims given above. It isn't recommended migrating from existing logging solutions to VictoriaLogs Preview in general case yet. @@ -35,6 +37,21 @@ vmagent (see [these docs](https://docs.victoriametrics.com/vmagent.html#how-to-c VictoriaLogs emits own logs to stdout. It is recommended investigating these logs during troubleshooting. +## Upgrading + +It is safe upgrading VictoriaLogs to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. +It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. +It is recommended performing regular upgrades to the latest version, since it may contain important bug fixes, performance optimizations or new features. + +It is also safe downgrading to older versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise. + +The following steps must be performed during the upgrade / downgrade procedure: + +* Send `SIGINT` signal to VictoriaLogs process in order to gracefully stop it. + See [how to send signals to processes](https://stackoverflow.com/questions/33239959/send-signal-to-process-from-command-line). +* Wait until the process stops. This can take a few seconds. +* Start the upgraded VictoriaMetrics. + ## Retention By default VictoriaLogs stores log entries with timestamps in the time range `[now-7d, now]`, while dropping logs outside the given time range. @@ -94,3 +111,14 @@ If `AccountID` and/or `ProjectID` request headers aren't set, then the default ` VictoriaLogs has very low overhead for per-tenant management, so it is OK to have thousands of tenants in a single VictoriaLogs instance. VictoriaLogs doesn't perform per-tenant authorization. Use [vmauth](https://docs.victoriametrics.com/vmauth.html) or similar tools for per-tenant authorization. + +## Benchmarks + +Here is a [benchmark suite](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/logs-benchmark) for comparing data ingestion performance +and resource usage between VictoriaLogs and Elasticsearch. + +It is recommended [setting up VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/QuickStart.html) in production alongside the existing +log management systems and comparing resource usage + query performance between VictoriaLogs and your system such as ElasticSearch or Grafana Loki. + +Please share benchmark results and ideas on how to improve benchmarks / VictoriaLogs +via [VictoriaMetrics community channels](https://docs.victoriametrics.com/#community-and-contributions). diff --git a/VictoriaLogs/Roadmap.md b/VictoriaLogs/Roadmap.md index dc3b845..799502b 100644 --- a/VictoriaLogs/Roadmap.md +++ b/VictoriaLogs/Roadmap.md @@ -17,8 +17,6 @@ The following functionality is planned in the future versions of VictoriaLogs: - Support for [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) from popular log collectors and formats: - Promtail (aka Grafana Loki) - - Vector.dev - - Fluentbit - Fluentd - Syslog - Add missing functionality to [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html): diff --git a/VictoriaLogs/data-ingestion/Filebeat.md b/VictoriaLogs/data-ingestion/Filebeat.md index 6793f7f..4bb1b58 100644 --- a/VictoriaLogs/data-ingestion/Filebeat.md +++ b/VictoriaLogs/data-ingestion/Filebeat.md @@ -72,7 +72,7 @@ output.elasticsearch: compression_level: 1 ``` -By default the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy). +By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy). If you need storing logs in other tenant, then specify the needed tenant via `headers` at `output.elasticsearch` section. For example, the following `filebeat.yml` config instructs Filebeat to store the data to `(AccountID=12, ProjectID=34)` tenant: @@ -88,6 +88,9 @@ output.elasticsearch: _stream_fields: "host.name,log.file.path" ``` -The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +See also: -See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs. +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +- [Filebeat `output.elasticsearch` docs](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html). +- [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/filebeat-docker). diff --git a/VictoriaLogs/data-ingestion/Fluentbit.md b/VictoriaLogs/data-ingestion/Fluentbit.md new file mode 100644 index 0000000..3d8ba73 --- /dev/null +++ b/VictoriaLogs/data-ingestion/Fluentbit.md @@ -0,0 +1,89 @@ +## Fluentbit setup + +Specify [http output](https://docs.fluentbit.io/manual/pipeline/outputs/http) section in the `fluentbit.conf` +for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/): + +```conf +[Output] + Name http + Match * + host localhost + port 9428 + uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 +``` + +Substitute the host (`localhost`) and port (`9428`) with the real TCP address of VictoriaLogs. + +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the query args specified in the `uri`. + +It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) +and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) in the `uri` +and inspecting VictoriaLogs logs then: + +```conf +[Output] + Name http + Match * + host localhost + port 9428 + uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1 + format json_lines + json_date_format iso8601 +``` + +If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: + +```conf +[Output] + Name http + Match * + host localhost + port 9428 + uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original + format json_lines + json_date_format iso8601 +``` + +If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option. +This usually allows saving network bandwidth and costs by up to 5 times: + +```conf +[Output] + Name http + Match * + host localhost + port 9428 + uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + compress gzip +``` + +By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#multitenancy). +If you need storing logs in other tenant, then specify the needed tenant via `header` options. +For example, the following `fluentbit.conf` config instructs Fluentbit to store the data to `(AccountID=12, ProjectID=34)` tenant: + +```conf +[Output] + Name http + Match * + host localhost + port 9428 + uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + header AccountID 12 + header ProjectID 23 +``` + +See also: + +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +- [Fluentbit HTTP output config docs](https://docs.fluentbit.io/manual/pipeline/outputs/http). +- [Docker-compose demo for Fluentbit integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker). diff --git a/VictoriaLogs/data-ingestion/Logstash.md b/VictoriaLogs/data-ingestion/Logstash.md index f527241..9e4fc45 100644 --- a/VictoriaLogs/data-ingestion/Logstash.md +++ b/VictoriaLogs/data-ingestion/Logstash.md @@ -74,7 +74,7 @@ output { } ``` -By default the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy). +By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy). If you need storing logs in other tenant, then specify the needed tenant via `custom_headers` at `output.elasticsearch` section. For example, the following `logstash.conf` config instructs Logstash to store the data to `(AccountID=12, ProjectID=34)` tenant: @@ -95,6 +95,9 @@ output { } ``` -The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +See also: -See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs. +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +- [Logstash `output.elasticsearch` docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html). +- [Docker-compose demo for Logstash integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash). diff --git a/VictoriaLogs/data-ingestion/README.md b/VictoriaLogs/data-ingestion/README.md index 9915820..5dbb852 100644 --- a/VictoriaLogs/data-ingestion/README.md +++ b/VictoriaLogs/data-ingestion/README.md @@ -3,11 +3,17 @@ [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can accept logs from the following log collectors: - Filebeat. See [how to setup Filebeat for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html). +- Fluentbit. See [how to setup Fluentbit for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html). - Logstash. See [how to setup Logstash for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html). +- Vector. See [how to setup Vector for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html). The ingested logs can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/). -See also [data ingestion troubleshooting](#troubleshooting) docs. +See also: + +- [Log collectors and data ingestion formats](#log-collectors-and-data-ingestion-formats). +- [Data ingestion troubleshooting](#troubleshooting). + ## HTTP APIs @@ -21,9 +27,10 @@ VictoriaLogs accepts optional [HTTP parameters](#http-parameters) at data ingest ### Elasticsearch bulk API VictoriaLogs accepts logs in [Elasticsearch bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) -format at `http://localhost:9428/insert/elasticsearch/_bulk` endpoint. +/ [OpenSearch Bulk API](http://opensearch.org/docs/1.2/opensearch/rest-api/document-apis/bulk/) format +at `http://localhost:9428/insert/elasticsearch/_bulk` endpoint. -The following command pushes a single log line to Elasticsearch bulk API at VictoriaLogs: +The following command pushes a single log line to VictoriaLogs: ```bash echo '{"create":{}} @@ -31,7 +38,14 @@ echo '{"create":{}} ' | curl -X POST -H 'Content-Type: application/json' --data-binary @- http://localhost:9428/insert/elasticsearch/_bulk ``` -The following command verifies that the data has been successfully pushed to VictoriaLogs by [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) it: +It is possible to push thousands of log lines in a single request to this API. + +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) for details on fields, +which must be present in the ingested log messages. + +The API accepts various http parameters, which can change the data ingestion behavior - [these docs](#http-parameters) for details. + +The following command verifies that the data has been successfully ingested to VictoriaLogs by [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) it: ```bash curl http://localhost:9428/select/logsql/query -d 'query=host.name:host123' @@ -43,10 +57,57 @@ The command should return the following response: {"_msg":"cannot open file","_stream":"{}","_time":"2023-06-21T04:24:24Z","host.name":"host123"} ``` +See also: + +- [How to debug data ingestion](#troubleshooting). +- [HTTP parameters, which can be passed to the API](#http-parameters). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying.html). + ### JSON stream API -TODO: document JSON stream API +VictoriaLogs accepts JSON line stream aka [ndjson](http://ndjson.org/) at `http://localhost:9428/insert/jsonline` endpoint. +The following command pushes multiple log lines to VictoriaLogs: + + ```bash +echo '{ "log": { "level": "info", "message": "hello world" }, "date": "2023-06-20T15:31:23Z", "stream": "stream1" } +{ "log": { "level": "error", "message": "oh no!" }, "date": "2023-06-20T15:32:10.567Z", "stream": "stream1" } +{ "log": { "level": "info", "message": "hello world" }, "date": "2023-06-20T15:35:11.567890+02:00", "stream": "stream2" } +' | curl -X POST -H 'Content-Type: application/stream+json' --data-binary @- \ + 'http://localhost:9428/insert/jsonline?_stream_fields=stream&_time_field=date&_msg_field=log.message' +``` + +It is possible to push unlimited number of log lines in a single request to this API. + +The [timestamp field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) must be +in the [ISO8601](https://en.wikipedia.org/wiki/ISO_8601) format. For example, `2023-06-20T15:32:10Z`. +Optional fractional part of seconds can be specified after the dot - `2023-06-20T15:32:10.123Z`. +Timezone can be specified instead of `Z` suffix - `2023-06-20T15:32:10+02:00`. + +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) for details on fields, +which must be present in the ingested log messages. + +The API accepts various http parameters, which can change the data ingestion behavior - [these docs](#http-parameters) for details. + +The following command verifies that the data has been successfully ingested into VictoriaLogs by [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) it: + +```bash +curl http://localhost:9428/select/logsql/query -d 'query=log.level:*' +``` + +The command should return the following response: + +```bash +{"_msg":"hello world","_stream":"{stream=\"stream2\"}","_time":"2023-06-20T13:35:11.56789Z","log.level":"info"} +{"_msg":"hello world","_stream":"{stream=\"stream1\"}","_time":"2023-06-20T15:31:23Z","log.level":"info"} +{"_msg":"oh no!","_stream":"{stream=\"stream1\"}","_time":"2023-06-20T15:32:10.567Z","log.level":"error"} +``` + +See also: + +- [How to debug data ingestion](#troubleshooting). +- [HTTP parameters, which can be passed to the API](#http-parameters). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying.html). ### HTTP parameters @@ -104,3 +165,14 @@ VictoriaLogs exposes various [metrics](https://docs.victoriametrics.com/Victoria since the last VictoriaLogs restart. If this metric grows rapidly during extended periods of time, then this may lead to [high cardinality issues](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#high-cardinality). The newly created log streams can be inspected in logs by passing `-logNewStreams` command-line flag to VictoriaLogs. + +## Log collectors and data ingestion formats + +Here is the list of log collectors and their ingestion formats supported by VictoriaLogs: + +| How to setup the collector | Format: Elasticsearch | Format: JSON Stream | +|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------------------------------------------------------| +| [Filebeat](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html) | [Yes](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) | No | +| [Fluentbit](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html) | No | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/http) | +| [Logstash](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html) | [Yes](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) | No | +| [Vector](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html) | [Yes](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) | No | diff --git a/VictoriaLogs/data-ingestion/Vector.md b/VictoriaLogs/data-ingestion/Vector.md new file mode 100644 index 0000000..9843d33 --- /dev/null +++ b/VictoriaLogs/data-ingestion/Vector.md @@ -0,0 +1,137 @@ +# Vector setup + +Specify [Elasticsearch sink type](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) in the `vector.toml` +for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/): + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" +``` + +Substitute the `localhost:9428` address inside `endpoints` section with the real TCP address of VictoriaLogs. + +Replace `your_input` with the name of the `inputs` section, which collects logs. See [these docs](https://vector.dev/docs/reference/configuration/sources/) for details. + +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on parameters specified +in the `[sinks.vlogs.query]` section. + +It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) +and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) +in the `[sinks.vlogs.query]` section and inspecting VictoriaLogs logs then: + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" + debug = "1" +``` + +If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" + ignore_fields = "log.offset,event.original" +``` + +When Vector ingests logs into VictoriaLogs at a high rate, then it may be needed to tune `batch.max_events` option. +For example, the following config is optimized for higher than usual ingestion rate: + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" + + [sinks.vlogs.batch] + max_events = 1000 +``` + +If the Vector sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compression = "gzip"` option. +This usually allows saving network bandwidth and costs by up to 5 times: + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + compression = "gzip" + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" +``` + +By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#multitenancy). +If you need storing logs in other tenant, then specify the needed tenant via `[sinks.vlogq.request.headers]` section. +For example, the following `vector.toml` config instructs Vector to store the data to `(AccountID=12, ProjectID=34)` tenant: + +```toml +[sinks.vlogs] + inputs = [ "your_input" ] + type = "elasticsearch" + endpoints = [ "http://localhost:9428/insert/elasticsearch/" ] + mode = "bulk" + api_version = "v8" + healthcheck.enabled = false + + [sinks.vlogs.query] + _msg_field = "message" + _time_field = "timestamp" + _stream_fields = "host,container_name" + + [sinks.vlogs.request.headers] + AccountID = "12" + ProjectID = "34" +``` + +See also: + +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). +- [Elasticsearch output docs for Vector](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/). +- [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-docker). diff --git a/VictoriaLogs/querying/README.md b/VictoriaLogs/querying/README.md index ec5ada4..de969af 100644 --- a/VictoriaLogs/querying/README.md +++ b/VictoriaLogs/querying/README.md @@ -1,6 +1,15 @@ # Querying -[VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can be queried at the `/select/logsql/query` endpoint. +[VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can be queried with [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) +via the following ways: + +- [Web UI](#web-ui) - a web-based UI for querying logs +- [HTTP API](#http-api) +- [Command-line interface](#command-line) + +## HTTP API + +VictoriaLogs can be queried at the `/select/logsql/query` HTTP endpoint. The [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) query must be passed via `query` argument. For example, the following query returns all the log entries with the `error` word: @@ -48,6 +57,30 @@ curl http://localhost:9428/select/logsql/query -H 'AccountID: 12' -H 'ProjectID: The number of requests to `/select/logsql/query` can be [monitored](https://docs.victoriametrics.com/VictoriaLogs/#monitoring) with `vl_http_requests_total{path="/select/logsql/query"}` metric. +## Web UI + +VictoriaLogs provides a simple Web UI for logs [querying](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) and exploration +at `http://localhost:9428/select/vmui`. The UI allows exploring query results: + + + +There are three modes of displaying query results: + +- `Group` - results are displayed as a table with rows grouped by stream and fields for filtering. +- `Table` - displays query results as a table. +- `JSON` - displays raw JSON response from [HTTP API](#http-api). + +This is the first version that has minimal functionality. It comes with the following limitations: + +- The number of query results is always limited to 1000 lines. Iteratively add + more specific [filters](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#filters) to the query + in order to get full response with less than 1000 lines. +- Queries are always executed against [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy) `0`. + +These limitations will be removed in future versions. + +To get around the current limitations, you can use an alternative - the [command line interface](#command-line). + ## Command-line VictoriaLogs integrates well with `curl` and other command-line tools during querying because of the following features: diff --git a/VictoriaLogs/querying/vmui.png b/VictoriaLogs/querying/vmui.png new file mode 100644 index 0000000..0032bcc Binary files /dev/null and b/VictoriaLogs/querying/vmui.png differ