Roman Khavronenko 5f33445f66 lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>		2022-06-07 15:02:55 +03:00
..
auth	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
awsapi	Add ability to sign requests for all AWS services (#2604)	2022-05-20 14:20:00 +03:00
backup	adds restore.lock (#1988)	2021-12-22 13:10:56 +02:00
blockcache	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293)	2022-03-26 12:57:27 +02:00
bloomfilter	lib/promscrape: add the ability to limit the number of unique series per each scrape target	2021-09-01 16:08:12 +03:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations	2022-02-01 11:20:20 +02:00
cgroup	lib/cgroup: reduce the default GOGC value from 50% to 30%	2022-04-06 14:00:50 +03:00
consts	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage	2022-04-05 15:39:58 +03:00
decimal	all: fix build issues and tests for Apple M1	2021-10-27 15:07:19 +03:00
encoding	lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint*	2022-04-12 12:56:52 +03:00
envflag	lib/envflag: use flag.Set for setting the flags from env vars	2021-10-20 00:46:53 +03:00
envtemplate	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components	2020-08-13 17:17:06 +03:00
fastnum	Makefile: add build and test rules with enabled race detector. These rules have -race suffix	2020-03-05 12:05:16 +02:00
fasttime	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
filestream	lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics	2021-12-02 09:13:20 +02:00
flagutil	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:58:08 +03:00
fs	app: allow specifying http and https urls in the following command-line flags	2021-12-03 00:11:47 +02:00
handshake	all: substitute zeroTime with time.Time{}, since this generates more optimal binary code	2022-02-07 14:36:41 +02:00
httpserver	lib/httpserver: clarify that -tls flag enables TLS for http requests to -httpListenAddr	2022-04-16 16:59:41 +03:00
influxutils	all: consistently return application/json content-type without charset=utf-8	2021-11-09 18:07:22 +02:00
ingestserver	lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig	2022-03-17 19:46:20 +02:00
leveledbytebufferpool	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics	2021-03-15 11:49:29 +02:00
logger	lib/logger: removed broken test after 746ee191e8	2022-01-24 12:15:11 +02:00
lrucache	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293)	2022-03-26 12:57:27 +02:00
memory	lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to	2022-04-07 15:24:08 +03:00
mergeset	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
metricsql	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:30:06 +03:00
netutil	lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get()	2022-05-11 14:11:06 +03:00
pacelimiter	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:33:07 +03:00
persistentqueue	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations	2022-02-01 11:20:20 +02:00
procutil	all: add go:build lines for Go1.17	2021-07-26 15:50:46 +03:00
promauth	lib/promauth: follow-up after 006b8c7534	2022-06-04 01:11:23 +03:00
prompb	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 22:53:50 +02:00
prompbmarshal	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
promrelabel	docs/CHANGELOG.md: follow-up after 2177089f94	2022-06-01 14:57:39 +03:00
promscrape	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string	2022-06-07 01:24:14 +03:00
promutils	lib/promscrape: properly implement ScrapeConfig.clone()	2022-05-07 00:06:19 +03:00
protoparser	lib/protoparser/native: follow-up after fe01f4803d	2022-04-11 19:27:53 +03:00
proxy	lib/promscrape: properly implement ScrapeConfig.clone()	2022-05-07 00:06:19 +03:00
querytracer	all: add initial support for query tracing	2022-06-01 02:31:44 +03:00
snapshot	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535)	2022-05-04 22:12:48 +03:00
storage	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673)	2022-06-07 15:02:55 +03:00
storagepacelimiter	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected	2020-07-30 20:02:22 +03:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration	2021-02-16 23:30:29 +02:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:30:10 +03:00
uint64set	Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method"	2021-07-06 18:26:56 +03:00
workingsetcache	lib/workingsetcache: reuse prev cache after its reset	2022-04-05 20:39:44 +03:00
writeconcurrencylimiter	fix: change thrashing typo (#2317)	2022-03-16 13:05:55 +02:00