VictoriaMetrics/docs/VictoriaLogs/data-ingestion/Fluentbit.md

weight	title	disableToc	menu	aliases
2	Fluentbit setup	true	docs parent weight victorialogs-data-ingestion 2	/VictoriaLogs/data-ingestion/Fluentbit.html /victorialogs/data-ingestion/fluentbit.html /victorialogs/data-ingestion/Fluentbit.html

Fluentbit setup

VictoriaLogs supports given below Fluentbit outputs:

• Elasticsearch
• Loki
• HTTP JSON

Elasticsearch

Specify elasticsearch output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[Output]
    Name es
    Match *
    host victorialogs
    port 9428
    compress gzip
    path /insert/elasticsearch
    header AccountID 0
    header ProjectID 0
    header VL-Stream-Fields path
    header VL-Msg-Field log
    header VL-Time-Field @timestamp

Loki

Specify loki output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[OUTPUT]
    name       loki
    match      *
    host       victorialogs
    uri        /insert/loki/api/v1/push
    port       9428
    label_keys $path,$log,$time
    header     VL-Msg-Field log
    header     VL-Time-Field time
    header     VL-Stream-Fields path

HTTP

Specify http output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601

Substitute the host (localhost) and port (9428) with the real TCP address of VictoriaLogs.

See these docs for details on the query args specified in the uri.

It is recommended verifying whether the initial setup generates the needed log fields and uses the correct stream fields. This can be done by specifying debug parameter in the uri and inspecting VictoriaLogs logs then:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1
     format json_lines
     json_date_format iso8601

If some log fields must be skipped during data ingestion, then they can be put into ignore_fields parameter. For example, the following config instructs VictoriaLogs to ignore log.offset and event.original fields in the ingested logs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original
     format json_lines
     json_date_format iso8601

If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via compress gzip option. This usually allows saving network bandwidth and costs by up to 5 times:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     compress gzip

By default, the ingested logs are stored in the (AccountID=0, ProjectID=0) tenant. If you need storing logs in other tenant, then specify the needed tenant via header options. For example, the following fluentbit.conf config instructs Fluentbit to store the data to (AccountID=12, ProjectID=34) tenant:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     header AccountID 12
     header ProjectID 23

See also:

• Data ingestion troubleshooting.
• How to query VictoriaLogs.
• Fluentbit HTTP output config docs.
• Docker-compose demo for Fluentbit integration with VictoriaLogs.