Aliaksandr Valialkin 0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores ... VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit 56b6b893ce . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244		2021-06-17 13:51:42 +03:00
..
auth	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
backup	lib/backup: increase backup chunk size from 128MB to 1GB	2021-01-13 12:16:39 +02:00
bloomfilter	lib/bloomfilter: fix TestLimiterConcurrent	2021-05-24 05:18:29 +03:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer	2020-08-05 11:55:12 +03:00
cgroup	lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for b50024812e	2021-05-13 09:27:35 +03:00
consts	all: open-sourcing cluster version	2019-05-23 00:25:38 +03:00
decimal	lib/decimal: prevent exponent overflow when processing values close to zero	2021-03-05 18:53:41 +02:00
encoding	lib: dump compressed block contents on error during decompression	2020-08-15 14:51:14 +03:00
envflag	lib/envflag: add -envflag.prefix for setting optional prefix for environment vars	2020-03-30 15:51:44 +03:00
envtemplate	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components	2020-08-13 17:17:06 +03:00
fastnum	Makefile: add build and test rules with enabled race detector. These rules have -race suffix	2020-03-05 12:05:16 +02:00
fasttime	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
filestream	fixes solaris build (#1345)	2021-06-04 11:56:06 +03:00
flagutil	improve docs for cli flags (#1202)	2021-04-12 12:28:36 +03:00
fs	fixes solaris build (#1345)	2021-06-04 11:56:06 +03:00
handshake	lib/handshake: pass io.EOF unmodified to the caller for BufferedConn.Read, so it could properly detect the end of stream	2021-05-27 12:09:17 +03:00
httpserver	app/vmauth: properly handle http.ErrAbortHandler panic	2021-06-11 12:54:37 +03:00
influxutils	lib/influxutils: return response compatible with InfluxDB 1.8.4	2021-03-15 22:20:38 +02:00
ingestserver	lib/ingestserver: properly close incoming connections during graceful shutdown	2021-05-08 19:53:45 +03:00
leveledbytebufferpool	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics	2021-03-15 11:49:29 +02:00
logger	improve docs for cli flags (#1202)	2021-04-12 12:28:36 +03:00
memory	fixes solaris build (#1345)	2021-06-04 11:56:06 +03:00
mergeset	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores	2021-06-17 13:51:42 +03:00
metricsql	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:30:06 +03:00
netutil	lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics	2021-03-17 00:19:30 +02:00
pacelimiter	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:33:07 +03:00
persistentqueue	lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value	2021-04-27 00:26:32 +03:00
procutil	Adds windows build (#1040)	2021-02-27 01:06:22 +02:00
promauth	lib/promauth: follow-up after 5b8176c68e	2021-05-22 18:02:03 +03:00
prompb	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 22:53:50 +02:00
prompbmarshal	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
promrelabel	lib/promrelabel: add tests for labelsToString() function	2021-06-04 20:42:14 +03:00
promscrape	lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages	2021-06-14 14:04:35 +03:00
protoparser	lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error	2021-06-14 15:20:38 +03:00
proxy	lib/promscrape: reload auth tokens from files every second	2021-05-14 20:03:35 +03:00
storage	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores	2021-06-17 13:51:42 +03:00
storagepacelimiter	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected	2020-07-30 20:02:22 +03:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration	2021-02-16 23:30:29 +02:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:30:10 +03:00
uint64set	lib/uint64set: store pointers to bucket16 instead of bucket16 objects in bucket32	2021-05-25 14:27:52 +03:00
workingsetcache	all: properly handle CPU limits set on the host system/container	2020-12-08 21:07:03 +02:00
writeconcurrencylimiter	all: properly handle CPU limits set on the host system/container	2020-12-08 21:07:03 +02:00