Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-19 07:01:02 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
8,387 Commits 114 Branches 748 Tags 596 MiB
85eee7de0a
Commit Graph

231 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
639967db59 lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions call. 
`runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions
can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction
deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction
deletions are finished before returning from `runTransactions`.
 2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin
b9616c017f lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed 
This should fix the issue on NFS when incompletely removed dirs may be left
after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction
files are already removed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin
26ffc77622 lib/{storage,mergeset}: create missing partition directories after restoring from backups 
Backup tools could skip empty directories. So re-create such directories on the first run.
 2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
5d2276dbf7 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:37 +02:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
9fce611fbb lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
 2019-09-30 12:27:30 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
3304dc1e85 lib/mergeset: detect whether we are in test by executable suffix 2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin
7d13c31566 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values 
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
 2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
bf8505353a lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts 2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin
ebbef20535 lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations 2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin
410f993bf6 lib/mergeset: fill partHeader.firstItem on first block flush 2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin
41ef6b060e lib/mergeset: properly check for sorted block headers 
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
 2019-09-13 21:59:38 +03:00
Aliaksandr Valialkin
568ff61dcf lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin
0b0153ba3d lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
 2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
604a4312f9 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
39f3f3a517 lib: move common code for creating flock.lock file into fs.CreateFlockFile 2019-08-13 01:46:20 +03:00
Aliaksandr Valialkin
73f866d874 lib/fs: atomically create file with the given contents on WriteFileAtomically 
This should prevent from `transaction` and `metadata.json` files corruption
on unclean shutdown such as OOM, `kill -9`, power loss, etc.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148
 2019-08-12 15:02:04 +03:00
Aliaksandr Valialkin
ba8195c58e all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
41f512af1c all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size 2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin
b6ea1a7d5e lib/mergeset: make fmt 2019-06-29 14:25:46 +03:00
Aliaksandr Valialkin
2257dcd278 lib/mergeset: speed up binarySearchKey by skipping the first item during binary search 2019-06-29 13:49:32 +03:00
Aliaksandr Valialkin
fb9358635d lib/storage: mention source parts on merge error 
This should improve determining broken source part.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
 2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin
18d6f293f7 lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func 
The func syncs parent dir in order to persist directory removal
in the event of power loss
 2019-06-12 01:55:18 +03:00
Aliaksandr Valialkin
51e2e255a6 lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error 2019-06-11 23:13:45 +03:00
Aliaksandr Valialkin
b491045a4b lib/{storage,mergeset}: sync filenames inside part when finalizing the part 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
 2019-06-11 21:51:19 +03:00
Aliaksandr Valialkin
3437c30180 all: try hard removing directory with contents 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
 2019-06-11 01:58:08 +03:00
Aliaksandr Valialkin
bdf696ef18 all: fix misspellings 2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00