VictoriaMetrics/app
Zhu Jiekun 3fa72b2c1b
feature: [victorialogs] drop logs without non-empty _msg field (#7056)
### Describe Your Changes

VictoriaLogs allows logs without `_msg` field or `_msg` field is empty.
This lead to incorrect search result. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785

This pull request search for non-empty `_msg` field before log entry is
added to `LogRows`.

New counter `vl_rows_dropped_total{reason="msg_not_exist"}` is
introduced.

Example log output:
```
2024-09-23T02:33:19.719Z        warn    app/vlinsert/insertutils/common_params.go:189   dropping log line without _msg field; [{@timestamp 2024-09-18T13:42:16.600000000Z} {Attributes.array.attribute ["many","values"]} {Attributes.boolean.attribute true} {Attributes.double.attribute 637.704} {Attributes.int.attribute 10} {Attributes.map.attribute.some.map.key some value} {Attributes.string.attribute some string} {Body Example ddddddddddlog record} {Resource.service.name my.service} {Scope.my.scope.attribute some scope attribute} {Scope.name my.library} {Scope.version 1.0.0} {SeverityNumber 10} {SeverityText Information} {SpanId eee19b7ec3c1b174} {TraceFlags 0} {TraceId 5b8efff798038103d269b633813fc60c}]
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] Benchmark for potential performance loss.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-26 09:35:58 +02:00
..
victoria-logs app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vlinsert feature: [victorialogs] drop logs without non-empty _msg field (#7056) 2024-09-26 09:35:58 +02:00
vlogsgenerator all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
vlselect app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop 2024-09-26 08:57:48 +02:00
vlstorage vendor: update github.com/VictoriaMetrics/metrics from v1.34.1 to v1.35.0 2024-07-15 10:45:39 +02:00
vmagent vmagent: remote write respect Retry-After in header (#6124) 2024-09-24 16:58:16 +02:00
vmalert app/vmalert: bump default values for sending data to remoteWrite.url (#7084) 2024-09-25 17:07:27 +02:00
vmalert-tool app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmauth app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmbackup app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmbackupmanager all: replace old https://docs.victoriametrics.com/vmbackupmanager.html url with the new one - https://docs.victoriametrics.com/vmbackupmanager/ 2024-04-18 02:04:39 +02:00
vmctl app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmgateway all: replace old https://docs.victoriametrics.com/vmgateway.html url with the new one - https://docs.victoriametrics.com/vmgateway/ 2024-04-18 02:08:53 +02:00
vminsert app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmrestore app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmselect app/vmselect/promql: consistently replace NaN data points with non-NaN values for range_first and range_last functions 2024-09-23 15:00:05 +02:00
vmstorage app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:15:22 +02:00
vmui vmui: optimize public directory by cleaning up files (#6972) 2024-09-19 15:48:10 +02:00