9c4b0334f2
The %q formatter may result in incorrectly formatted JSON string if the original string
contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string
cannot be parsed by JSON parsers.
This is a follow-up for
|
||
|---|---|---|
| .. | ||
| main.go | ||
| Makefile | ||
| README.md | ||
vlogsgenerator
Logs generator for VictoriaLogs.
How to build vlogsgenerator?
Run make vlogsgenerator from the repository root. This builds bin/vlogsgenerator binary.
How run vlogsgenerator?
vlogsgenerator generates logs in JSON line format suitable for the ingestion
via /insert/jsonline endpoint at VictoriaLogs.
By default it writes the generated logs into stdout. For example, the following command writes generated logs to stdout:
bin/vlogsgenerator
It is possible to redirect the generated logs to file. For example, the following command writes the generated logs to logs.json file:
bin/vlogsgenerator > logs.json
The generated logs at logs.json file can be inspected with the following command:
head logs.json | jq .
Below is an example output:
{
"_time": "2024-05-08T14:34:00.854Z",
"_msg": "message for the stream 8 and worker 0; ip=185.69.136.129; uuid=b4fe8f1a-c93c-dea3-ba11-5b9f0509291e; u64=8996587920687045253",
"host": "host_8",
"worker_id": "0",
"run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
"const_0": "some value 0 8",
"const_1": "some value 1 8",
"const_2": "some value 2 8",
"var_0": "some value 0 12752539384823438260",
"dict_0": "warn",
"dict_1": "info",
"u8_0": "6",
"u16_0": "35202",
"u32_0": "1964973739",
"u64_0": "4810489083243239145",
"float_0": "1.868",
"ip_0": "250.34.75.125",
"timestamp_0": "1799-03-16T01:34:18.311Z",
"json_0": "{\"foo\":\"bar_3\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":32}"
}
{
"_time": "2024-05-08T14:34:00.854Z",
"_msg": "message for the stream 9 and worker 0; ip=164.244.254.194; uuid=7e8373b1-ce0d-1ce7-8e96-4bcab8955598; u64=13949903463741076522",
"host": "host_9",
"worker_id": "0",
"run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
"const_0": "some value 0 9",
"const_1": "some value 1 9",
"const_2": "some value 2 9",
"var_0": "some value 0 5371555382075206134",
"dict_0": "INFO",
"dict_1": "FATAL",
"u8_0": "219",
"u16_0": "31459",
"u32_0": "3918836777",
"u64_0": "6593354256620219850",
"float_0": "1.085",
"ip_0": "253.151.88.158",
"timestamp_0": "2042-10-05T16:42:57.082Z",
"json_0": "{\"foo\":\"bar_5\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":27}"
}
The run_id field uniquely identifies every vlogsgenerator invocation.
How to write logs to VictoriaLogs?
The generated logs can be written directly to VictoriaLogs by passing the address of /insert/jsonline endpoint
to -addr command-line flag. For example, the following command writes the generated logs to VictoriaLogs running at localhost:
bin/vlogsgenerator -addr=http://localhost:9428/insert/jsonline
Configuration
vlogsgenerator accepts various command-line flags, which can be used for configuring the number and the shape of the generated logs.
These flags can be inspected by running vlogsgenerator -help. Below are the most interesting flags:
-start- starting timestamp for generating logs. Logs are evenly generated on the [-start...-end] interval.-end- ending timestamp for generating logs. Logs are evenly generated on the [-start...-end] interval.-activeStreams- the number of active log streams to generate.-logsPerStream- the number of log entries to generate per each log stream. Log entries are evenly distributed on the [-start...-end] interval.
The total number of generated logs can be calculated as -activeStreams * -logsPerStream.
For example, the following command generates 1_000_000 log entries on the time range [2024-01-01 - 2024-02-01] across 100
log streams, where every logs stream contains 10_000 log entries,
and writes them to http://localhost:9428/insert/jsonline:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
Churn rate
It is possible to generate churn rate for active log streams
by specifying -totalStreams command-line flag bigger than -activeStreams. For example, the following command generates
logs for 1000 total streams, while the number of active streams equals to 100. This means that at every time there are logs for 100 streams,
but these streams change over the given [-start ... -end] time range, so the total number of streams on the given time range becomes 1000:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-totalStreams=1_000 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
In this case the total number of generated logs equals to -totalStreams * -logsPerStream = 10_000_000.
Benchmark tuning
By default vlogsgenerator generates and writes logs by a single worker. This may limit the maximum data ingestion rate during benchmarks.
The number of workers can be changed via -workers command-line flag. For example, the following command generates and writes logs with 16 workers:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-workers=16
Output statistics
Every 10 seconds vlogsgenerator writes statistics about the generated logs into stderr. The frequency of the generated statistics can be adjusted via -statInterval command-line flag.
For example, the following command writes statistics every 2 seconds:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-statInterval=2s