Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin
9dc9c892b7
lib/encoding: optimizing UnmarshalVarUint64 and UnmarshalVarInt64 a bit
2024-05-12 16:35:24 +02:00
Aliaksandr Valialkin
87338633b1
lib/slicesutil: add helper functions for setting slice length and extending its capacity
...
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
2024-05-12 11:33:49 +02:00
Aliaksandr Valialkin
d845edc24b
lib: consistently use atomic.* types instead of atomic.* functions
...
See ea9e2b19a5
2024-02-24 02:10:04 +02:00
Aliaksandr Valialkin
faeabfc730
lib/encoding: follow-up for 49e3665d6d
...
Improve performance for typical cases of varint marshaling / unmarshaling further.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
2024-02-01 05:38:58 +02:00
Fuchun Zhang
78af9b3e30
make encoding.MarshalVarInt64s faster ( #5721 )
...
* make encoding.MarshalVarInt64s faster
* add fast path for MarshalVarInt64s
* make UnmarshalVarUint64s faster
* remove comment
2024-02-01 03:33:59 +00:00
Aliaksandr Valialkin
eee210810e
lib/encoding: added benchmarks for marshaling / unmarshaling of varints
...
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
2024-02-01 05:11:35 +02:00
Aliaksandr Valialkin
8dd73574ca
lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint*
2024-01-23 22:50:11 +02:00
Aliaksandr Valialkin
a45cbc101f
all: cleanup: remove // +build ...
lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20
2023-11-13 19:15:42 +01:00
Aliaksandr Valialkin
dcf5b42670
lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...
...
The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer.
This is a follow-up for 140e7b6b74
2023-07-20 21:54:51 -07:00
Aliaksandr Valialkin
992c300ce9
all: replace atomic.Value with atomic.Pointer[T]
...
This eliminates the need in .(*T) casting for results obtained from Load()
Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map,
because map is already a pointer type.
2023-07-19 17:48:26 -07:00
Aliaksandr Valialkin
4c80193a86
lib/encoding: add MarshalBool/UnmarshalBool and GetUint32s/PutUint32s functions
...
These functions are going to be used by VictoriaLogs
2023-07-06 17:24:52 -07:00
Aliaksandr Valialkin
bf545fcc14
lib/encoding: fix test after 4725549cb2
2023-04-05 21:38:48 -07:00
Aliaksandr Valialkin
dfb1d1ead1
lib/encoding: make deterministic tests which rely on math/rand
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
2023-01-23 18:43:49 -08:00
Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
...
This reduces CPU usage when there is no sense in validating timestamps.
This is a follow-up for 5fa9525498
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
051e722112
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:07:49 +03:00
Aliaksandr Valialkin
4a3172f150
lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint*
...
This allows Go complier to generate more optimal code without bound checks
2022-04-12 12:56:52 +03:00
Aliaksandr Valialkin
084664d780
lib/encoding: substitute 64-bits.LeadingZeros64()
with bits.Len64()
2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
5d255846ac
all: add go:build
lines for Go1.17
...
See https://tip.golang.org/doc/go1.17#gofmt for more details
2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin
edf3ee2a9b
lib: dump compressed block contents on error during decompression
...
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
2aecf7c37c
lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting
2020-04-28 00:02:19 +03:00
Aliaksandr Valialkin
806dc73d8a
lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize
2020-04-28 00:02:18 +03:00
Aliaksandr Valialkin
d3c185f0ca
lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled
...
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin
091e35cf0c
lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta*
...
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin
0e51058a0d
lib/encoding: mention unpacked block size in the error message if unparsed tail left
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin
44f886cc9c
lib/encoding/zstd: typo fix
2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin
108a60d69e
lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines
...
This should fix goroutine leak for https://github.com/klauspost/compress/issues/195
2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin
335bd0ac0a
lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel
...
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin
5d439cc6f2
lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones
...
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
1037053fed
lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones
...
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
272e2f77c9
lib/encoding: optimize UnmarshalUint* and UnmarshalInt*
2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
f2e8d54fb0
lib/encoding/zstd: go fmt
2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin
97b5dc7122
lib/encoding/zstd: disable CRC checks in pure Go
build
...
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin
54f035d4ce
all: small updates after PR #114
2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin
7a133567fb
lib/encoding: small fixes in tests after the PR #114
2019-07-24 17:43:39 +03:00
Roman Khavronenko
fcf09aaa3c
all: add Pure Go build (pull request #114 )
...
Updates #94
2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin
3fae34eeb4
lib/encoding: improve gauge series detection
...
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets
This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin
bdf696ef18
all: fix misspellings
2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
9b64dfee4b
lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total
for determining the number CompressZSTD / DecompressZSTD calls
2019-05-24 13:01:15 +03:00
Aliaksandr Valialkin
b23352dc9e
lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total
metrics for rough estimation of block compression ratio
2019-05-24 12:34:51 +03:00
Aliaksandr Valialkin
f67f40d63a
lib/encoding: substitute CompressZSTD with CompressZSTDLevel
2019-05-24 12:32:49 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00