VictoriaMetrics/lib
Roman Khavronenko 5f33445f66
lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673)
Workers count for merges affects the max part size during merges. Such behaviour
protects storage from running out of disk space for scenario when all workers
are merging parts with the max size.

This works very well for most cases. But for systems where high number of CPUs
is allocated for vmstorage components this could significantly impact the max
part size and result in more unmerged parts than expected.

While checking multiple production highly loaded setups it was discovered that
`max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2,
and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4.
The change in this commit limits the max value for concurrency accordingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-07 15:02:55 +03:00
..
auth all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:33:46 +03:00
awsapi Add ability to sign requests for all AWS services (#2604) 2022-05-20 14:20:00 +03:00
backup adds restore.lock (#1988) 2021-12-22 13:10:56 +02:00
blockcache lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293) 2022-03-26 12:57:27 +02:00
bloomfilter lib/promscrape: add the ability to limit the number of unique series per each scrape target 2021-09-01 16:08:12 +03:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 2022-02-01 11:20:20 +02:00
cgroup lib/cgroup: reduce the default GOGC value from 50% to 30% 2022-04-06 14:00:50 +03:00
consts app/vminsert: reduce the max packet size, which vminsert can send to vmstorage 2022-04-05 15:39:58 +03:00
decimal all: fix build issues and tests for Apple M1 2021-10-27 15:07:19 +03:00
encoding lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* 2022-04-12 12:56:52 +03:00
envflag lib/envflag: use flag.Set for setting the flags from env vars 2021-10-20 00:46:53 +03:00
envtemplate all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 2020-08-13 17:17:06 +03:00
fastnum Makefile: add build and test rules with enabled race detector. These rules have -race suffix 2020-03-05 12:05:16 +02:00
fasttime lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
filestream lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics 2021-12-02 09:13:20 +02:00
flagutil lib/flagutil: re-use Duration.Set() call in NewDuration 2022-05-02 10:58:08 +03:00
fs app: allow specifying http and https urls in the following command-line flags 2021-12-03 00:11:47 +02:00
handshake all: substitute zeroTime with time.Time{}, since this generates more optimal binary code 2022-02-07 14:36:41 +02:00
httpserver lib/httpserver: clarify that -tls flag enables TLS for http requests to -httpListenAddr 2022-04-16 16:59:41 +03:00
influxutils all: consistently return application/json content-type without charset=utf-8 2021-11-09 18:07:22 +02:00
ingestserver lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig 2022-03-17 19:46:20 +02:00
leveledbytebufferpool lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics 2021-03-15 11:49:29 +02:00
logger lib/logger: removed broken test after 746ee191e8 2022-01-24 12:15:11 +02:00
lrucache lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293) 2022-03-26 12:57:27 +02:00
memory lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to 2022-04-07 15:24:08 +03:00
mergeset lib/storage: stop background merge when storage enters read-only mode 2022-06-01 14:22:12 +03:00
metricsql lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:30:06 +03:00
netutil lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() 2022-05-11 14:11:06 +03:00
pacelimiter lib/pacelimiter: increase scalability for multi-CPU system 2020-08-06 18:33:07 +03:00
persistentqueue lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 2022-02-01 11:20:20 +02:00
procutil all: add go:build lines for Go1.17 2021-07-26 15:50:46 +03:00
promauth lib/promauth: follow-up after 006b8c7534 2022-06-04 01:11:23 +03:00
prompb app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps 2020-01-28 22:53:50 +02:00
prompbmarshal all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:33:46 +03:00
promrelabel docs/CHANGELOG.md: follow-up after 2177089f94 2022-06-01 14:57:39 +03:00
promscrape lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string 2022-06-07 01:24:14 +03:00
promutils lib/promscrape: properly implement ScrapeConfig.clone() 2022-05-07 00:06:19 +03:00
protoparser lib/protoparser/native: follow-up after fe01f4803d 2022-04-11 19:27:53 +03:00
proxy lib/promscrape: properly implement ScrapeConfig.clone() 2022-05-07 00:06:19 +03:00
querytracer all: add initial support for query tracing 2022-06-01 02:31:44 +03:00
snapshot {vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535) 2022-05-04 22:12:48 +03:00
storage lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) 2022-06-07 15:02:55 +03:00
storagepacelimiter lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 2020-07-30 20:02:22 +03:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration 2021-02-16 23:30:29 +02:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:30:10 +03:00
uint64set Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" 2021-07-06 18:26:56 +03:00
workingsetcache lib/workingsetcache: reuse prev cache after its reset 2022-04-05 20:39:44 +03:00
writeconcurrencylimiter fix: change thrashing typo (#2317) 2022-03-16 13:05:55 +02:00