From a0ff5c2627aead668bf9cdae4ec85c325a562c26 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Mon, 10 Jul 2023 15:00:10 -0700 Subject: [PATCH] docs/VictoriaLogs: make more prominent the information about returned log fields in query responses Thanks to @candlerb for suggestions on how to improve VictoriaLogs docs at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4609#issuecomment-1629758426 --- docs/VictoriaLogs/LogsQL.md | 24 +++++++++++++++++++++- docs/VictoriaLogs/data-ingestion/README.md | 10 +++++++++ docs/VictoriaLogs/querying/README.md | 5 +++++ 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/VictoriaLogs/LogsQL.md b/docs/VictoriaLogs/LogsQL.md index 2509b80c4..35b758a1e 100644 --- a/docs/VictoriaLogs/LogsQL.md +++ b/docs/VictoriaLogs/LogsQL.md @@ -78,7 +78,8 @@ The query returns the following [log fields](https://docs.victoriametrics.com/Vi - [`_time` field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) Logs may contain arbitrary number of other fields. If you need obtaining some of these fields in query results, -then just refer them in the query with `field_name:*` [filter](#any-value-filter). +then just refer them in the query with `field_name:*` [filter](#any-value-filter). See [these docs](#querying-specific-fields) for more details. + For example, the following query returns `host.hostname` field additionally to `_msg`, `_stream` and `_time` fields: ```logsql @@ -1088,6 +1089,27 @@ Additionally, LogsQL will provide the ability to select fields, which must be re See the [Roadmap](https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html) for details. +## Querying specific fields + +By default VictoriaLogs query response contains [`_msg`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field), +[`_stream`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) and +[`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields. + +If you want selecting other fields from the ingested [structured logs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model), +then they must be mentioned in query filters. For example, if you want selecting `log.level` field, and this field isn't mentioned in the query yet, then add +`log.level:*` [filter](#any-value-filter) filter to the end of the query. +The `field_name:*` filter doesn't return log entries with empty or missing `field_name`. If you want returning log entries +with and without the given field, then `(field_name:* OR field_name:"")` filter can be used. +See the following docs for details: + +- [Any value filter](#any-value-filter) +- [Empty value filter](#empty-value-filter) +- [Logical filter](#logical-filter) + +In the future LogsQL will support `| fields field1, field2, ... fieldN` syntax for selecting the listed fields. +It will also support the ability to select all the fields for the matching log entries with `| fields *` syntax. +See the [Roadmap](https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html) for details. + ## Performance tips - It is highly recommended specifying [time filter](#time-filter) in order to narrow down the search to specific time range. diff --git a/docs/VictoriaLogs/data-ingestion/README.md b/docs/VictoriaLogs/data-ingestion/README.md index 5dbb85203..5faeef494 100644 --- a/docs/VictoriaLogs/data-ingestion/README.md +++ b/docs/VictoriaLogs/data-ingestion/README.md @@ -57,6 +57,11 @@ The command should return the following response: {"_msg":"cannot open file","_stream":"{}","_time":"2023-06-21T04:24:24Z","host.name":"host123"} ``` +Note that the response contains [`_msg`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field), +[`_stream`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) and +[`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields. +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details. + See also: - [How to debug data ingestion](#troubleshooting). @@ -103,6 +108,11 @@ The command should return the following response: {"_msg":"oh no!","_stream":"{stream=\"stream1\"}","_time":"2023-06-20T15:32:10.567Z","log.level":"error"} ``` +Note that the response contains [`_msg`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field), +[`_stream`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) and +[`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields. +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details. + See also: - [How to debug data ingestion](#troubleshooting). diff --git a/docs/VictoriaLogs/querying/README.md b/docs/VictoriaLogs/querying/README.md index de969af84..1fded3c76 100644 --- a/docs/VictoriaLogs/querying/README.md +++ b/docs/VictoriaLogs/querying/README.md @@ -17,6 +17,11 @@ For example, the following query returns all the log entries with the `error` wo curl http://localhost:9428/select/logsql/query -d 'query=error' ``` +Note that the response contains [`_msg`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field), +[`_stream`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) and +[`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields. +See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details. + The `query` argument can be passed either in the request url itself (aka HTTP GET request) or via request body with the `x-www-form-urlencoded` encoding (aka HTTP POST request). The HTTP POST is useful for sending long queries when they do not fit the maximum url length of the used clients and proxies.