Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 20:37:12 +01:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity
9,485 Commits 153 Branches 748 Tags 386 MiB
11af902b22
Commit Graph

27 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
b3d9d36fb3
lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 
See ea9e2b19a5
 2024-02-24 00:15:26 +02:00
Aliaksandr Valialkin
2eb967231b
lib/storage: do not append headerData to bsw.indexData if its size exceeds maxBlockSize 
This is a follow-up optimization after 3c246cdf00
 2024-02-12 18:04:37 +02:00
Aliaksandr Valialkin
3c246cdf00
lib/{storage,mergeset}: do not create index blocks with sizes exceeding 64Kb in common case 
This should reduce memory fragmentation and memory usage for indexdb/indexBlocks and storage/indexBlocks caches
 2024-02-08 13:52:24 +02:00
Aliaksandr Valialkin
c0b852d50d
lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 
Callers of InitFromFilePart log the error and exit.
It is better to log the error with the path to the part and the call stack
directly inside the MustInitFromFilePart() function.
This simplifies the code at callers' side while leaving the same level of debuggability.
 2023-04-14 15:46:12 -07:00
Aliaksandr Valialkin
9183a439c7
lib/filestream: change Create() to MustCreate() 
Callers of this function log the returned error and exit.
It is better logging the error together with the path to the filename
and call stack directly inside the function. This simplifies
the code at callers' side without reducing the level of debuggability
 2023-04-14 15:12:48 -07:00
Aliaksandr Valialkin
bd6de6406a
lib/fs: improve error logging inside MustWriteData 
Log the path to file on errors inside MustWriteData().
This improves debuggability of errors, which may occur inside MustWriteData().
 2023-04-14 14:32:45 -07:00
Aliaksandr Valialkin
036a7b7365
lib/fs: replace MkdirAllIfNotExist->MustMkdirIfNotExist and MkdirAllFailIfExist->MustMkdirFailIfExist 
Callers of these functions log the returned error and then exit. The returned error already contains the path
to directory, which was failed to be created. So let's just log the error together with the call stack
inside these functions. This leaves the debuggability of the returned error at the same level
while allows simplifying the code at callers' side.

While at it, properly use MustMkdirFailIfExist instead of MustMkdirIfNotExist inside inmemoryPart.MustStoreToDisk().
It is expected that the inmemoryPart.MustStoreToDick() must fail if there is already a directory under the given path.
 2023-04-13 22:11:59 -07:00
Aliaksandr Valialkin
b15c5961ab
lib/{mergeset,storage}: remove unused path field from blockStreamWriter 
This is a follow-up after 42bba64aa7
 2023-04-13 21:39:59 -07:00
Aliaksandr Valialkin
42bba64aa7
lib/{mergeset,storage}: explicitly fsync the created part directory listing 
Previously the created part directory listing was fsynced implicitly
when storing metadata.json file in it.

Also remove superflouous fsync for part directory listing,
which was called at blockStreamWriter.MustClose().
After that the metadata.json file is created, so an additional fsync
for the directory contents is needed.
 2023-04-13 21:03:08 -07:00
Aliaksandr Valialkin
c8f2febaa1
lib/storage: consistently use OS-independent separator in file paths 
This is needed for Windows support, which uses `\` instead of `/` as file separator

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
 2023-03-25 14:33:58 -07:00
Oleksandr Redko
9fff48c3e3
app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
Aliaksandr Valialkin
343c69fc15
lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 
This allows packing in-memory blocks with different compression levels
depending on its contents. This may save memory usage.
 2022-12-03 22:46:48 -08:00
Aliaksandr Valialkin
042a532f70
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
 2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
4ff647137a
lib/storage: deduplicate samples more thoroughly 
Previously some duplicate samples may be left on disk for time series with high churn rate.
This may result in higher disk space usage.
 2021-12-15 15:59:58 +02:00
Aliaksandr Valialkin
92070cbb67
lib/storage: return dedup interval in milliseconds from GetDedupInterval() 
This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.
 2021-12-15 13:26:38 +02:00
Aliaksandr Valialkin
1d20a19c7d
lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() 
This improves the code readability and debuggability, since the output of these functions
stops depending on global state.
 2021-12-14 20:49:12 +02:00
John Belmonte
364fdf4a56
spelling fix: adjacent (#1115) 2021-03-09 09:18:19 +02:00
Aliaksandr Valialkin
fda61e8e96 lib/storage: skip deduplication when creating inmemory data blocks 
The deduplication will be performed later during merging such blocks.
 2021-02-09 02:25:32 +02:00
Aliaksandr Valialkin
9d8fdff6c5 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
0eacea1de1 lib/{storage,mergeset}: further tuning of compression levels depending on block size 
This should improve performance for querying newly added data, since it can be unpacked faster.
 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
c3d86eef96 all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
9becc26f4b lib/storage: remove interface conversion in hot path during block merging 
This should improve merge speed a bit for parts with big number of small blocks.
 2019-11-03 12:33:34 +02:00
Aliaksandr Valialkin
935bfd7a18 lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error 2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin
20fc0e0e54 lib/{storage,mergeset}: sync filenames inside part when finalizing the part 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
 2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin
ac7b186f13 all: try hard removing directory with contents 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
 2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00