VictoriaMetrics/docs/VictoriaLogs/data-ingestion/Logstash.md

weight	title	disableToc	menu	aliases
3	Logstash setup	true	docs parent weight victorialogs-data-ingestion 3	/VictoriaLogs/data-ingestion/Logstash.html /victorialogs/data-ingestion/logstash.html /victorialogs/data-ingestion/Logstash.html

VictoriaLogs supports given below Logstash outputs:

• Elasticsearch
• Loki
• HTTP JSON

Elasticsearch

Specify output.elasticsearch section in the logstash.conf file for sending the collected logs to VictoriaLogs:

output {
  elasticsearch {
    hosts => ["http://localhost:9428/insert/elasticsearch/"]
    parameters => {
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
        "_stream_fields" => "host.name,process.name"
    }
  }
}

Substitute localhost:9428 address inside hosts with the real TCP address of VictoriaLogs.

See these docs for details on the parameters section.

It is recommended verifying whether the initial setup generates the needed log fields and uses the correct stream fields. This can be done by specifying debug parameter and inspecting VictoriaLogs logs then:

output {
  elasticsearch {
    hosts => ["http://localhost:9428/insert/elasticsearch/"]
    parameters => {
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
        "_stream_fields" => "host.name,process.name"
        "debug" => "1"
    }
  }
}

If some log fields must be skipped during data ingestion, then they can be put into ignore_fields parameter. For example, the following config instructs VictoriaLogs to ignore log.offset and event.original fields in the ingested logs:

output {
  elasticsearch {
    hosts => ["http://localhost:9428/insert/elasticsearch/"]
    parameters => {
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
        "_stream_fields" => "host.hostname,process.name"
        "ignore_fields" => "log.offset,event.original"
    }
  }
}

If the Logstash sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via http_compression: true option. This usually allows saving network bandwidth and costs by up to 5 times:

output {
  elasticsearch {
    hosts => ["http://localhost:9428/insert/elasticsearch/"]
    parameters => {
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
        "_stream_fields" => "host.hostname,process.name"
    }
    http_compression => true
  }
}

By default, the ingested logs are stored in the (AccountID=0, ProjectID=0) tenant. If you need storing logs in other tenant, then specify the needed tenant via custom_headers at output.elasticsearch section. For example, the following logstash.conf config instructs Logstash to store the data to (AccountID=12, ProjectID=34) tenant:

output {
  elasticsearch {
    hosts => ["http://localhost:9428/insert/elasticsearch/"]
    custom_headers => {
        "AccountID" => "1"
        "ProjectID" => "2"
    }
    parameters => {
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
        "_stream_fields" => "host.hostname,process.name"
    }
  }
}

Loki

Specify output.loki section in the logstash.conf file for sending the collected logs to VictoriaLogs:

output {
  loki {
     url => "http://victorialogs:9428/insert/loki/api/v1/push?_stream_fields=host.ip,process.name&_msg_field=message&_time_field=@timestamp"
  }
}

HTTP

Specify output.http section in the logstash.conf file for sending the collected logs to VictoriaLogs:

output {
  url => "http://victorialogs:9428/insert/jsonline?_stream_fields=host.ip,process.name&_msg_field=message&_time_field=@timestamp"
  format => "json"
  http_method => "post"
}

See also:

• Data ingestion troubleshooting.
• How to query VictoriaLogs.
• Logstash output.elasticsearch docs.
• Docker-compose demo for Logstash integration with VictoriaLogs.