22 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	bced9ee666	lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute ... This should reduce memory usage on a system with high number of active time series and a high churn rate. One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).	2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin	34195218e1	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	9566015a36	Revert "lib/mergeset: tune lifetime for entries inside block caches" ... This reverts commit 458c89324d. Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.	2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin	35a23234ca	lib/mergeset: tune lifetime for entries inside block caches ... This should reduce memory usage in general case without significant CPU usage increase	2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin	e8ee9fa7fe	app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches	2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin	7f3e884a31	all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9	2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin	f4fd917e4f	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin	c736339843	lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively ... Previously such blocks were cleaned after they weren't accessed during 10 minutes. Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.	2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin	d962568e93	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin	3845420a8f	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin	f442d81648	lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse	2020-05-14 14:03:15 +03:00
Aliaksandr Valialkin	bd5f4e0344	lib/storage: properly initialize part struct before trying to close it on error ... This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .	2020-05-12 14:54:16 +03:00
Aliaksandr Valialkin	0b2f678d8e	lib/{storage,mergeset}: make sure that requests and misses cache counters never go down	2020-04-10 14:44:52 +03:00
Aliaksandr Valialkin	347aaba79d	lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate ... It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin	da19fffa08	all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	62b041e90a	lib/{mergeset,storage}: properly update lastAccessTime in index and data block cache entries	2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin	caffb0cd01	lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)	2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin	b03ccbf6f7	lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches ... This should reduce memory usage in the long run when old blocks and indexes aren't accessed anymore.	2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin	5b01b7fb01	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	0a8dd9cc9a	lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent ... This should improve query speed over recent data on machines with big amounts of RAM	2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin	41f512af1c	all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size	2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00