mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-15 16:30:55 +01:00
82 lines
2.6 KiB
Markdown
82 lines
2.6 KiB
Markdown
|
# Benchmark for VictoriaLogs
|
||
|
|
|
||
|
|
Benchmark is based on:
|
||
|
|
|
||
|
|
- Logs from this repository - https://github.com/logpai/loghub
|
||
|
|
- filebeat - https://www.elastic.co/beats/filebeat
|
||
|
|
- elastic + kibana
|
||
|
|
- [logs generator](./generator)
|
||
|
|
|
||
|
|
## How it works
|
||
|
|
|
||
|
|
docker-compose.yml contains:
|
||
|
|
|
||
|
|
- 2 filebeat instances - one for elastic and one for VictoriaLogs.
|
||
|
|
- elastic instance
|
||
|
|
- VictoriaLogs instance
|
||
|
|
- kibana instance - port forwarded to `localhost:5601` to see UI
|
||
|
|
- vmsingle - port forwarded to `localhost:8428` to see UI
|
||
|
|
- [logs generator](./generator)
|
||
|
|
- exporters for filebeat/system
|
||
|
|
|
||
|
|
[Logs generator](./generator) generates logs based on logs located at `./source_logs/logs` and sends them to filebeat
|
||
|
|
instances via syslog.
|
||
|
|
Logs are generated by reading files line by line, adding randomized suffix to each line and sending them to filebeat via
|
||
|
|
syslog.
|
||
|
|
By default, generator will exit once all files are read. `docker-compose` will restart it and files will be read again
|
||
|
|
generating new logs.
|
||
|
|
|
||
|
|
Each filebeat than writes logs to elastic and VictoriaLogs via elasticsearch-compatible API.
|
||
|
|
|
||
|
|
## How to run
|
||
|
|
|
||
|
|
1. Download and unarchive logs by running:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
cd source_logs
|
||
|
|
bash download.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
Note that with logs listed in `download.sh` it will require 49GB of free space:
|
||
|
|
|
||
|
|
- 3GB for archives
|
||
|
|
- 46GB for unarchived logs
|
||
|
|
|
||
|
|
If it is needed to minimize disk footprint, you can download only some of them by commenting out lines in `download.sh`.
|
||
|
|
Unarchived logs size per file for reference:
|
||
|
|
```shell
|
||
|
|
2.3M Linux.log
|
||
|
|
73M SSH.log
|
||
|
|
32G Thunderbird.log
|
||
|
|
5.1M Apache.log
|
||
|
|
13G hadoop-*.log
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
2. (optional) If needed, adjust amount of logs sent by generator by modifying `-outputRateLimitItems` and
|
||
|
|
`outputRateLimitPeriod` parameters in [docker-compose.yml](./docker-compose.yml). By default, it is configured to
|
||
|
|
send 10000 logs per second.
|
||
|
|
|
||
|
|
3. Build victoria-logs image and adjust `image` parameter in [docker-compose.yml](./docker-compose.yml):
|
||
|
|
|
||
|
|
```shell
|
||
|
|
make package-victoria-logs
|
||
|
|
```
|
||
|
|
|
||
|
|
Image name should be replaced at `vlogs` service in [docker-compose.yml](./docker-compose.yml).
|
||
|
|
|
||
|
|
It is also possible to configure filebeat to send logs to VictoriaLogs running on local machine.
|
||
|
|
To do this modify [filebeat config for vlogs](./elk/filebeat/filebeat-vlogs.yml) and replace `vlogs` address
|
||
|
|
with address of local VictoriaLogs instance:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
output.elasticsearch:
|
||
|
|
hosts: [ "http://vlogs:9428/insert/elasticsearch/" ]
|
||
|
|
```
|
||
|
|
|
||
|
|
4. Run `docker-compose up -d` to start benchmark.
|
||
|
|
|
||
|
|
5. Navigate to `http://localhost:3000/d/hkm6P6_4z/elastic-vs-vlogs` to see Grafana dashboard with resource usage comparison.
|
||
|
|
|
||
|
|
|