VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin f548adce0b
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 16:48:21 -07:00
..
arena.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_data_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_data.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_search.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_stream_merger.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_stream_reader.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_stream_writer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
consts.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
datadb.go lib/logstorage: fix panic (#4620) 2023-07-13 09:53:41 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filters_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filters.go lib/logstorage: LogsQL: replace exact_prefix("...") with exact("..."*) 2023-07-17 19:07:40 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
log_rows.go app/vlinsert/loki: follow-up after 09df5b66fd 2023-07-20 16:48:21 -07:00
parser_test.go lib/logstorage: go fmt after a8000b74c5 2023-07-18 16:04:51 -07:00
parser.go lib/logstorage: properly encode "offset" search word just after _time filter 2023-07-18 16:00:06 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go app/victoria-logs: add ability to debug data ingestion by passing debug query arg to data ingestion API 2023-06-20 20:02:46 -07:00
rows_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
rows.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage_search_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage_search.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage.go lib/logstorage: log the -retentionPeriod and -futureRetention values when the ingested log entry has timestamp outside the configured retention 2023-07-17 19:07:41 -07:00
stream_filter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_tags.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tenant_id_test.go app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 10:10:55 +02:00
tenant_id.go app/vlinsert/loki: follow-up after 09df5b66fd 2023-07-20 16:48:21 -07:00
tokenizer_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder_test.go lib/logstorage: fix TestValuesEncoder() on 32-bit architectures 2023-07-13 11:27:13 -07:00
values_encoder_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00