Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 16:30:55 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
3,521 Commits 114 Branches 748 Tags 504 MiB
9aa9b081a4
Commit Graph

91 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
ab4be24397
app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches 
The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query:

   vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9
 2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin
b885bd9b7d lib/{mergeset,storage}: improve the detection of the needed free space for background merge 
This should prevent from possible out of disk space crashes during big merges.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560
 2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin
51cd19d2e3 lib/storage: reset perKeyMisses stats less frequently 
This should reduce CPU usage for queries executed with intervals higher than 30 seconds
 2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin
41754e12f8 lib/mergeset: cache indexBlock items only on the second request 
This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.
 2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin
9826f7c1be lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) 
This should reduce the cache size and memory usage for the indexdb/dataBlocks cache
 2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin
b805a675f3 lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects 
This should reduce memory usage on systems with big number of CPU cores,
since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains
a separate pool inmemoryPart objects per each CPU core.

Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized
by big sizes of inmemoryPart objects.
 2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin
db6bd69475 lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload 
CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small.
This results in the increased load on memory allocation code in Go runtime.
Increase the pool capacity in order to reduce the load on Go runtime.
 2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin
fd32855a6c lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds 
This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered
(aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.
 2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin
f12f97daa1 lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes 
One minute cache timeout result in slower queries in some production workloads where the interval
between query execution is in the range 1 minute - 2 minutes.
 2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin
8055439fe4 lib/storage: properly detect free disk space shortage during data merge 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373
 2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin
bced9ee666 lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute 
This should reduce memory usage on a system with high number of active time series and a high churn rate.
One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).
 2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin
b7c0b3dde3 lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs 
This should reduce memory usage for the pool on systems with big number of CPU cores.

The sync.Pool maintains per-CPU pools, so the total number of objects in the pool
is proportional to the number of available CPU cores. The channel limits the number
of pooled objects by its own capacity. This means smaller number of pooled objects on average.
 2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin
0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin
6865f3b497 Revert "lib/mergeset: remove a pool for inmemoryBlock structs" 
This reverts commit 793fe39921.

Reason to revert: production testing revealed possible slowdown when registering big number of new time series
 2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin
7b33bc67a1 lib/mergeset: remove a pool for inmemoryBlock structs 
The pool for inmemoryBlock struct doesn't give any performance gains in production workloads,
while it may result in excess memory usage for inmemoryBlock structs inside the pool during
background merge of indexdb.
 2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin
0fc857d363 lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds 
Log these messages if the merge takes more than 30 seconds instead of 10 seconds.
 2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
e37e1b1e34 lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures 
The panic has been introduced in 56b6b893ce
 2021-04-27 16:42:19 +03:00
Aliaksandr Valialkin
2d1d60118d lib/mergeset: split rows ingestion among multiple shards 
This improves rows ingestion on systems with many CPU cores by reducing lock contention.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244

Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243
 2021-04-27 15:45:11 +03:00
Aliaksandr Valialkin
ab8008d6d7 lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
 2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin
587132555f lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin
b8a5ee2e93 lib/{mergeset,storage}: allow merging smaller number of small parts 
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.

This partially reverts ebf8da3730
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
34195218e1 lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
9566015a36 Revert "lib/mergeset: tune lifetime for entries inside block caches" 
This reverts commit 458c89324d.

Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.
 2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin
35a23234ca lib/mergeset: tune lifetime for entries inside block caches 
This should reduce memory usage in general case without significant CPU usage increase
 2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin
500acb958c lib/mergeset: clarify comments in the code a bit 2021-02-16 18:03:11 +02:00
Aliaksandr Valialkin
d788876a7a lib/mergeset: remove unused code after a4140de9e6 2021-02-16 13:40:29 +02:00
Aliaksandr Valialkin
afa9cf9c57 lib/mergeset: remove dead code left after a4140de9e6 2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
e8ee9fa7fe app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:49:58 +02:00
faceair
cf76d8dd79 lib/mergeset: add missing shouldCacheBlock (#1019) 2021-01-15 11:47:25 +02:00
Aliaksandr Valialkin
edbe35509e lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage 2020-12-18 20:01:20 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
8f8727cb65 lib/mergeset: tune the number of rawItemsBlocks to merge at once 
512 blocks give higher ingestion performance and slightly lower memory usage
 2020-11-25 21:53:15 +02:00
Aliaksandr Valialkin
8fcd87a6a5 lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset 2020-11-25 21:20:02 +02:00
Aliaksandr Valialkin
7f3e884a31 all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9 2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin
f4fd917e4f lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin
c736339843 lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively 
Previously such blocks were cleaned after they weren't accessed during 10 minutes.
Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.
 2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin
3abbb38254 lib/{mergeset,storage}: compare errors with errors.Is() 2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin
ddb3519e17 lib/{mergeset,storage}: code prettifying 2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin
edf3ee2a9b lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
29bbab0ec9 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
b8303afcd8 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
6afdcf8a20 lib/mergeset: properly calculate global metrics in UpdateStats() 
Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.
 2020-07-23 00:35:29 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
a72f18e821 lib/{storage,mergeset}: further tuning of compression levels depending on block size 
This should improve performance for querying newly added data, since it can be unpacked faster.
 2020-05-15 13:12:28 +03:00
Aliaksandr Valialkin
6838fa876c lib/mergeset: tune compression levels in order to improve ingestion performance a bit 2020-05-15 12:12:15 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
7e831741f9 lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused 2020-05-14 15:32:23 +03:00
Aliaksandr Valialkin
f442d81648 lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse 2020-05-14 14:03:15 +03:00
Aliaksandr Valialkin
0b2f678d8e lib/{storage,mergeset}: make sure that requests and misses cache counters never go down 2020-04-10 14:44:52 +03:00