Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 09:15:26 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
7,212 Commits 114 Branches 748 Tags 513 MiB
d7a63529b5
Commit Graph

198 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
ea7940e5a7
lib/mergeset: reduce the parts threshold before starting assisted merges 
This should improve query speed in general case.

This is a follow-up for d1af6046c7
 2022-12-13 09:14:08 -08:00
Aliaksandr Valialkin
2a190f6451
lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead 
Blocked small merges may result into big number of small parts, which, in turn,
may result in increased CPU and memory usage during queries, since queries need to inspect
all the existing small parts.

The issue has been introduced in 8189770c50

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
 2022-12-12 17:01:33 -08:00
Aliaksandr Valialkin
0a9992a9c6
lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown 2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin
7d5c64eb7a
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk 
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
 2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin
9ac1174493
lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function 2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin
a13d21513e
lib/mergeset: panic when too long item is passed to Table.AddItems() 2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin
bb93494eac
lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers 
This simplifies the code
 2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin
ea55c16422
lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 
This allows packing in-memory blocks with different compression levels
depending on its contents. This may save memory usage.
 2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin
eca7f32151
lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part 
Previously only data was compressed with the given compressLevel
 2022-12-03 22:35:16 -08:00
Aliaksandr Valialkin
7ffa66d249
lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part 
This results in more correct reporting of memory usage for in-memory parts
 2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin
886ce94739
lib/mergeset: reduce the time needed for the slowest tests 2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin
10a17bfa16
lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items} 2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin
0b8e7deabd
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
 2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin
d8d4d21d7a
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts 
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
 2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin
b0b8f05fa4
lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() 
The issue has been introduced in 58b40f514c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
 2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin
06300fe9b8
lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf 
Previously indexBlock.bhs pointed to the buffer, which could be changed over time.
This could result in incorrect time series search over time.

This is a follow-up for 58b40f514c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
 2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin
454060fd78
lib/mergeset: remove string allocation and copying when unmarshaling blockHeader 
This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
 2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
1fb2be0cae
lib/{mergeset,storage}: simplify the code a bit after ae55ad8749 2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin
6855de311c
lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms 
The panic has been introduced in 68f3a02589

While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs

This should improve scalability on systems with many CPU cores
 2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin
68f3a02589
lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list 2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin
7a6e5f9224
lib/mergeset: mention in the error message the path to the part, which triggered the error 
This should improve debuggability
 2022-10-12 09:54:42 +03:00
Roman Khavronenko
fe71c73fe1
lib/mergeset: follow-up after a0e7432e42 (#3145) 
* lib/mergeset: follow-up after a0e7432e42

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-26 16:43:17 +03:00
Aliaksandr Valialkin
5b488a339d
lib/mergeset: atomically remove part dirs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
 2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll 
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
 2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
db049fed84
lib/mergeset: cleanup after de6dd1cd5a 
Remove unused getInmemoryPart and putInmemoryPart functions

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
 2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin
962ed46583
lib/mergeset: optimize mergeInmemoryBlocks() function 
Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs.
Just merge inmemoryBlock structs directly.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
 2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin
3bbe9054d3
lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge 
Just read the current item directly from blockStreamReader.Block.Items
with the helper method - blockStreamReader.CurrItem()
 2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin
547cb1edce
benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254

This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
 2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin
5f2b5bd173
lib/mergeset: add tests and benchmarks for commonPrefixLen function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254

This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
 2022-07-27 21:25:23 +03:00
zhenyuxie
14c6212a61
fix inmemoryBlock's Less method (#2881) 2022-07-18 12:00:45 +03:00
Dmytro Kozlov
5256af2291
lib/mergeset: fix linter error (#2864) 2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin
7cbcbea49d
lib/mergeset: optimize merge speed a bit 
Use heap.Fix instead of heap.Pop + heap.Push when merging blocks
 2022-07-12 12:52:36 +03:00
Loki's Wager
ca4730c00f
BugFix part_header.go (#2763) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757

Co-authored-by: haotingyi <haotingyi@corp.netease.com>
 2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin
fedfc9e686
lib/storage: stop background merge when storage enters read-only mode 
This should prevent from `no space left on device` errors when VictoriaMetrics
under-estimates the additional disk space needed for background merge.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603
 2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin
ed1b394a1a
app/vmstorage: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page 
These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index).

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471
 2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin
edb139cfe4
lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to 
This metric is equivalent to `vm_available_memory_bytes`, but it has better name,
since the metric is related to a process, not VictoriaMetrics itself.

Leave `vm_available_memory_bytes` for backwards compatibility.
 2022-04-07 15:24:08 +03:00
Aliaksandr Valialkin
8ef9348801
lib/mergeset: use more rawItemsShard shards on multi-CPU systems 
This should improve the scalability for registering of new time series on multi-CPU system
 2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin
db00ddd23e
lib/mergeset: skip common prefixes when comparing inmemoryBlock items 
This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData
if they have common prefix.

While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items.
This should improve performance for inmemoryBlock.MarshalSortedData during background merge.
 2022-04-06 18:55:25 +03:00
Aliaksandr Valialkin
3f999b11f7
lib/mergeset: remove aux buffers from inmemoryPart 
This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
 2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
ecf68da79e
lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock 
This should improve performance when registering new time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
 2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
ecf4f7bf21
lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars 2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
f4e466955d
lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool 
The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart,
so sync.Pool usage should reduce overall memory usage and improve performance
because of better locality of reference when marshaling inmemoryBlock to inmemoryPart.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
 2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
b47f18f555
lib/{mergeset,storage}: tune compression levels for small blocks 
This should reduce CPU usage spent on compression
 2022-02-25 15:34:13 +02:00
Aliaksandr Valialkin
1967b9c211
lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort() 
There is no need to sort the underlying data according to sorted items there.
This should reduce cpu usage when registering new time series in `indexdb`.

Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245
 2022-02-24 11:19:44 +02:00
Aliaksandr Valialkin
986c12487a
lib/mergeset: typo fix after b6ed9afd6d 2022-02-21 19:00:04 +02:00
Roman Khavronenko
bd7837d524
lib: allow to configure cache size by type (#2206) 
* lib: allow to configure cache size by type

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-21 13:55:51 +02:00
Aliaksandr Valialkin
895c9f4f11
lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin
a0a56d6c1c
lib/mergeset: tune sizes for indexdb/dataBlocks and indexdb/indexBlocks according to production workload 
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742
 2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin
d8d59ff760
lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
02b2bfcff3
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 
Follow-up for f4989edd96

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 11:20:20 +02:00
匠心零度
a1083d0531
optimized code (#2103) 
* optimized code ,because only the first error,so no need var errors []error

* optimized code ,because only the first error,so no need var errors []error

Co-authored-by: lirenzuo <lirenzuo@shein.com>
 2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin
6232eaa938
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions 
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().

Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.

Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin
7ec0705b98
lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock 
This should reduce the memory required for indexdb/dataBlocks cache.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin
9edf407144
lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin
6ae584b9b3
lib/{mergeset,storage}: properly limit cache sizes for indexdb 
Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`,
since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded,
which could result to out of memory errors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin
ab4be24397
app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches 
The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query:

   vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9
 2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin
b885bd9b7d lib/{mergeset,storage}: improve the detection of the needed free space for background merge 
This should prevent from possible out of disk space crashes during big merges.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560
 2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin
51cd19d2e3 lib/storage: reset perKeyMisses stats less frequently 
This should reduce CPU usage for queries executed with intervals higher than 30 seconds
 2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin
41754e12f8 lib/mergeset: cache indexBlock items only on the second request 
This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.
 2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin
9826f7c1be lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) 
This should reduce the cache size and memory usage for the indexdb/dataBlocks cache
 2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin
b805a675f3 lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects 
This should reduce memory usage on systems with big number of CPU cores,
since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains
a separate pool inmemoryPart objects per each CPU core.

Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized
by big sizes of inmemoryPart objects.
 2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin
db6bd69475 lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload 
CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small.
This results in the increased load on memory allocation code in Go runtime.
Increase the pool capacity in order to reduce the load on Go runtime.
 2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin
fd32855a6c lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds 
This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered
(aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.
 2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin
f12f97daa1 lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes 
One minute cache timeout result in slower queries in some production workloads where the interval
between query execution is in the range 1 minute - 2 minutes.
 2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin
8055439fe4 lib/storage: properly detect free disk space shortage during data merge 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373
 2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin
bced9ee666 lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute 
This should reduce memory usage on a system with high number of active time series and a high churn rate.
One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).
 2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin
b7c0b3dde3 lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs 
This should reduce memory usage for the pool on systems with big number of CPU cores.

The sync.Pool maintains per-CPU pools, so the total number of objects in the pool
is proportional to the number of available CPU cores. The channel limits the number
of pooled objects by its own capacity. This means smaller number of pooled objects on average.
 2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin
0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin
6865f3b497 Revert "lib/mergeset: remove a pool for inmemoryBlock structs" 
This reverts commit 793fe39921.

Reason to revert: production testing revealed possible slowdown when registering big number of new time series
 2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin
7b33bc67a1 lib/mergeset: remove a pool for inmemoryBlock structs 
The pool for inmemoryBlock struct doesn't give any performance gains in production workloads,
while it may result in excess memory usage for inmemoryBlock structs inside the pool during
background merge of indexdb.
 2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin
0fc857d363 lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds 
Log these messages if the merge takes more than 30 seconds instead of 10 seconds.
 2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
e37e1b1e34 lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures 
The panic has been introduced in 56b6b893ce
 2021-04-27 16:42:19 +03:00
Aliaksandr Valialkin
2d1d60118d lib/mergeset: split rows ingestion among multiple shards 
This improves rows ingestion on systems with many CPU cores by reducing lock contention.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244

Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243
 2021-04-27 15:45:11 +03:00
Aliaksandr Valialkin
ab8008d6d7 lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
 2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin
587132555f lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin
b8a5ee2e93 lib/{mergeset,storage}: allow merging smaller number of small parts 
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.

This partially reverts ebf8da3730
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
34195218e1 lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
9566015a36 Revert "lib/mergeset: tune lifetime for entries inside block caches" 
This reverts commit 458c89324d.

Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.
 2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin
35a23234ca lib/mergeset: tune lifetime for entries inside block caches 
This should reduce memory usage in general case without significant CPU usage increase
 2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin
500acb958c lib/mergeset: clarify comments in the code a bit 2021-02-16 18:03:11 +02:00
Aliaksandr Valialkin
d788876a7a lib/mergeset: remove unused code after a4140de9e6 2021-02-16 13:40:29 +02:00
Aliaksandr Valialkin
afa9cf9c57 lib/mergeset: remove dead code left after a4140de9e6 2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
e8ee9fa7fe app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:49:58 +02:00
faceair
cf76d8dd79 lib/mergeset: add missing shouldCacheBlock (#1019) 2021-01-15 11:47:25 +02:00
Aliaksandr Valialkin
edbe35509e lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage 2020-12-18 20:01:20 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
8f8727cb65 lib/mergeset: tune the number of rawItemsBlocks to merge at once 
512 blocks give higher ingestion performance and slightly lower memory usage
 2020-11-25 21:53:15 +02:00
Aliaksandr Valialkin
8fcd87a6a5 lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset 2020-11-25 21:20:02 +02:00
Aliaksandr Valialkin
7f3e884a31 all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9 2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin
f4fd917e4f lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin
c736339843 lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively 
Previously such blocks were cleaned after they weren't accessed during 10 minutes.
Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.
 2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin
3abbb38254 lib/{mergeset,storage}: compare errors with errors.Is() 2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin
ddb3519e17 lib/{mergeset,storage}: code prettifying 2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin
edf3ee2a9b lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
29bbab0ec9 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
b8303afcd8 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
6afdcf8a20 lib/mergeset: properly calculate global metrics in UpdateStats() 
Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.
 2020-07-23 00:35:29 +03:00