mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2025-01-20 07:19:17 +01:00
update wiki pages
Vika
parent
00f1f2168f
commit
f4b97e2024
@ -1047,7 +1047,7 @@ VictoriaMetrics provides the following handlers for exporting data:
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export?match[]=<timeseries_selector_for_export>`,
|
||||
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
for metrics to export. Use `{__name__!=""}` selector for fetching all the time series.
|
||||
The response would contain all the data for the selected time series in [JSON streaming format](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON).
|
||||
The response would contain all the data for the selected time series in [JSON streaming format](http://ndjson.org/).
|
||||
Each JSON line contains samples for a single time series. An example output:
|
||||
|
||||
```json
|
||||
|
||||
@ -1055,7 +1055,7 @@ VictoriaMetrics provides the following handlers for exporting data:
|
||||
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export?match[]=<timeseries_selector_for_export>`,
|
||||
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
for metrics to export. Use `{__name__!=""}` selector for fetching all the time series.
|
||||
The response would contain all the data for the selected time series in [JSON streaming format](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON).
|
||||
The response would contain all the data for the selected time series in [JSON streaming format](http://ndjson.org/).
|
||||
Each JSON line contains samples for a single time series. An example output:
|
||||
|
||||
```json
|
||||
|
||||
@ -82,32 +82,14 @@ It is recommended setting up monitoring of VictoriaLogs according to [these docs
|
||||
|
||||
### Data ingestion
|
||||
|
||||
VictoriaLogs supports the following data ingestion techniques:
|
||||
VictoriaLogs supports the following data ingestion approaches:
|
||||
|
||||
- Via [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html). See [these docs](#filebeat-setup).
|
||||
- Via [Logstash](https://www.elastic.co/guide/en/logstash/current/introduction.html). See [these docs](#logstash-setup).
|
||||
|
||||
The ingested log entries can be queried according to [these docs](#querying).
|
||||
The ingested logs can be queried according to [these docs](#querying).
|
||||
|
||||
#### Data ingestion troubleshooting
|
||||
|
||||
VictoriaLogs provides the following command-line flags, which can help debugging data ingestion issues:
|
||||
|
||||
- `-logNewStreams` - if this flag is passed to VictoriaLogs, then it logs all the newly
|
||||
registered [log streams](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This may help debugging [high cardinality issues](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#high-cardinality).
|
||||
- `-logIngestedRows` - if this flag is passed to VictoriaLogs, then it logs all the ingested
|
||||
[log entries](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model).
|
||||
|
||||
VictoriaLogs exposes various [metrics](#monitoring), which may help debugging data ingestion issues:
|
||||
|
||||
- `vl_rows_ingested_total` - the number of ingested [log entries](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
since the last VictoriaLogs restart. If this number icreases over time, then logs are successfully ingested into VictoriaLogs.
|
||||
The ingested logs can be inspected in logs by passing `-logIngestedRows` command-line flag to VictoriaLogs.
|
||||
- `vl_streams_created_total` - the number of created [log streams](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields)
|
||||
since the last VictoriaLogs restart. If this metric grows rapidly during extended periods of time, then this may lead
|
||||
to [high cardinality issues](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#high-cardinality).
|
||||
The newly created log streams can be inspected in logs by passing `-logNewStreams` command-line flag to VictoriaLogs.
|
||||
See also [data ingestion troubleshooting](#data-ingestion-troubleshooting) docs.
|
||||
|
||||
#### Filebeat setup
|
||||
|
||||
@ -125,17 +107,24 @@ output.elasticsearch:
|
||||
|
||||
Substitute the `localhost:9428` address inside `hosts` section with the real TCP address of VictoriaLogs.
|
||||
|
||||
The `_msg_field` parameter must contain the field name with the log message generated by Filebeat. This is usually `message` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) for details.
|
||||
See [these docs](#data-ingestion-parameters) for details on the `parameters` section.
|
||||
|
||||
The `_time_field` parameter must contain the field name with the log timestamp generated by Filebeat. This is usually `@timestamp` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) for details.
|
||||
It is recommended to verify whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This can be done by specifying `debug` [parameter](#data-ingestion-parameters):
|
||||
|
||||
It is recommended specifying comma-separated list of field names, which uniquely identify every log stream collected by Filebeat, in the `_stream_fields` parameter.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details.
|
||||
```yml
|
||||
output.elasticsearch:
|
||||
hosts: ["http://localhost:9428/insert/elasticsearch/"]
|
||||
parameters:
|
||||
_msg_field: "message"
|
||||
_time_field: "@timestamp"
|
||||
_stream_fields: "host.hostname,log.file.path"
|
||||
debug: "1"
|
||||
```
|
||||
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) aren't needed,
|
||||
then VictoriaLogs can be instructed to ignore them during data ingestion - just pass `ignore_fields` parameter with comma-separated list of fields to ignore.
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped
|
||||
during data ingestion, then they can be put into `ignore_fields` [parameter](#data-ingestion-parameters).
|
||||
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
|
||||
|
||||
```yml
|
||||
@ -215,17 +204,28 @@ output {
|
||||
|
||||
Substitute `localhost:9428` address inside `hosts` with the real TCP address of VictoriaLogs.
|
||||
|
||||
The `_msg_field` parameter must contain the field name with the log message generated by Logstash. This is usually `message` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) for details.
|
||||
See [these docs](#data-ingestion-parameters) for details on the `parameters` section.
|
||||
|
||||
The `_time_field` parameter must contain the field name with the log timestamp generated by Logstash. This is usually `@timestamp` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) for details.
|
||||
It is recommended to verify whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This can be done by specifying `debug` [parameter](#data-ingestion-parameters):
|
||||
|
||||
It is recommended specifying comma-separated list of field names, which uniquely identify every log stream collected by Logstash, in the `_stream_fields` parameter.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details.
|
||||
```conf
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
||||
parameters => {
|
||||
"_msg_field" => "message"
|
||||
"_time_field" => "@timestamp"
|
||||
"_stream_fields" => "host.name,process.name"
|
||||
"debug" => "1"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) aren't needed,
|
||||
then VictoriaLogs can be instructed to ignore them during data ingestion - just pass `ignore_fields` parameter with comma-separated list of fields to ignore.
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped
|
||||
during data ingestion, then they can be put into `ignore_fields` [parameter](#data-ingestion-parameters).
|
||||
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
|
||||
|
||||
```conf
|
||||
@ -284,6 +284,55 @@ The ingested log entries can be queried according to [these docs](#querying).
|
||||
|
||||
See also [data ingestion troubleshooting](#data-ingestion-troubleshooting) docs.
|
||||
|
||||
#### Data ingestion parameters
|
||||
|
||||
VictoriaLogs accepts the following parameters at [data ingestion](#data-ingestion) HTTP APIs:
|
||||
|
||||
- `_msg_field` - it must contain the name of the [log field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
with the [log message](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) generated by the log shipper.
|
||||
This is usually the `message` field for Filebeat and Logstash.
|
||||
If the `_msg_field` parameter isn't set, then VictoriaLogs reads the log message from the `_msg` field.
|
||||
|
||||
- `_time_field` - it must contain the name of the [log field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
with the [log timestamp](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) generated by the log shipper.
|
||||
This is usually the `@timestamp` field for Filebeat and Logstash.
|
||||
If the `_time_field` parameter isn't set, then VictoriaLogs reads the timestamp from the `_time` field.
|
||||
If this field doesn't exist, then the current timestamp is used.
|
||||
|
||||
- `_stream_fields` - it should contain comma-separated list of [log field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) names,
|
||||
which uniquely identify every [log stream](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) collected the log shipper.
|
||||
If the `_stream_fields` parameter isn't set, then all the ingested logs are written to default log stream - `{}`.
|
||||
|
||||
- `ignore_fields` - this parameter may contain the list of [log field](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) names,
|
||||
which must be ignored during [data ingestion](#data-ingestion).
|
||||
|
||||
- `debug` - if this parameter is set to `1`, then the [ingested](#data-ingestion) logs aren't stored in VictoriaLogs. Instead,
|
||||
the ingested data is logged by VictoriaLogs, so it can be investigated later.
|
||||
|
||||
#### Data ingestion troubleshooting
|
||||
|
||||
VictoriaLogs provides the following command-line flags, which can help debugging data ingestion issues:
|
||||
|
||||
- `-logNewStreams` - if this flag is passed to VictoriaLogs, then it logs all the newly
|
||||
registered [log streams](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This may help debugging [high cardinality issues](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#high-cardinality).
|
||||
- `-logIngestedRows` - if this flag is passed to VictoriaLogs, then it logs all the ingested
|
||||
[log entries](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model).
|
||||
See also `debug` [parameter](#data-ingestion-parameters).
|
||||
|
||||
VictoriaLogs exposes various [metrics](#monitoring), which may help debugging data ingestion issues:
|
||||
|
||||
- `vl_rows_ingested_total` - the number of ingested [log entries](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
since the last VictoriaLogs restart. If this number icreases over time, then logs are successfully ingested into VictoriaLogs.
|
||||
The ingested logs can be inspected in the following ways:
|
||||
- By passing `debug=1` parameter to every request to [data ingestion endpoints](#data-ingestion). The ingested rows aren't stored in VictoriaLogs
|
||||
in this case. Instead, they are logged, so they can be investigated later. The `vl_rows_dropped_total` [metric](#monitoring) is incremented for each logged row.
|
||||
- By passing `-logIngestedRows` command-line flag to VictoriaLogs. In this case it logs all the ingested data, so it can be investigated later.
|
||||
- `vl_streams_created_total` - the number of created [log streams](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields)
|
||||
since the last VictoriaLogs restart. If this metric grows rapidly during extended periods of time, then this may lead
|
||||
to [high cardinality issues](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#high-cardinality).
|
||||
The newly created log streams can be inspected in logs by passing `-logNewStreams` command-line flag to VictoriaLogs.
|
||||
|
||||
### Querying
|
||||
|
||||
VictoriaLogs can be queried at the `/select/logsql/query` endpoint. The [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html)
|
||||
@ -468,12 +517,12 @@ outside the configured retention.
|
||||
|
||||
VictoriaLogs automatically drops logs at [data ingestion](#data-ingestion) stage if they have timestamps outside the configured retention.
|
||||
A sample of dropped logs is logged with `WARN` message in order to simplify troubleshooting.
|
||||
The `vlinsert_rows_dropped_total` [metric](#monitoring) is incremented each time an ingested log entry is dropped because of timestamp outside the retention.
|
||||
The `vl_rows_dropped_total` [metric](#monitoring) is incremented each time an ingested log entry is dropped because of timestamp outside the retention.
|
||||
It is recommended setting up the following alerting rule at [vmalert](https://docs.victoriametrics.com/vmalert.html) in order to be notified
|
||||
when logs with wrong timestamps are ingested into VictoriaLogs:
|
||||
|
||||
```metricsql
|
||||
rate(vlinsert_rows_dropped_total[5m]) > 0
|
||||
rate(vl_rows_dropped_total[5m]) > 0
|
||||
```
|
||||
|
||||
By default VictoriaLogs doesn't accept log entries with timestamps bigger than `now+2d`, e.g. 2 days in the future.
|
||||
|
||||
@ -174,7 +174,7 @@ Single-node VictoriaMetrics:
|
||||
<div class="with-copy" markdown="1">
|
||||
|
||||
```console
|
||||
curl --data-binary "@filename.json" -X POST http://localhost:8428/api/v1/import
|
||||
curl -H 'Content-Type: application/json' --data-binary "@filename.json" -X POST http://localhost:8428/api/v1/import
|
||||
```
|
||||
|
||||
</div>
|
||||
@ -183,7 +183,7 @@ Cluster version of VictoriaMetrics:
|
||||
<div class="with-copy" markdown="1">
|
||||
|
||||
```console
|
||||
curl --data-binary "@filename.json" -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import
|
||||
curl -H 'Content-Type: application/json' --data-binary "@filename.json" -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import
|
||||
```
|
||||
|
||||
</div>
|
||||
@ -497,7 +497,7 @@ echo '
|
||||
}
|
||||
]
|
||||
}
|
||||
' | curl -X POST --data-binary @- http://localhost:8428/datadog/api/v1/series
|
||||
' | curl -X POST -H 'Content-Type: application/json' --data-binary @- http://localhost:8428/datadog/api/v1/series
|
||||
```
|
||||
|
||||
</div>
|
||||
@ -524,7 +524,7 @@ echo '
|
||||
}
|
||||
]
|
||||
}
|
||||
' | curl -X POST --data-binary @- 'http://<vminsert>:8480/insert/0/datadog/api/v1/series'
|
||||
' | curl -X POST -H 'Content-Type: application/json' --data-binary @- 'http://<vminsert>:8480/insert/0/datadog/api/v1/series'
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user