Commit Graph

424 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
33ea2120c3 lib/storage: use unversioned keys for tag cache in extDB
Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache.
This should improve performance for index lookups over big amount of time series.
2019-06-25 13:08:58 +03:00
Aliaksandr Valialkin
cf63669303 lib/storage: skip searching in extDB if it doesn't contain items for the given time range
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
2019-06-25 13:00:37 +03:00
Aliaksandr Valialkin
af2ceaaa0b lib/storage: mention source parts on merge error
This should improve determining broken source part.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
2019-06-24 14:08:43 +03:00
Aliaksandr Valialkin
80db24386e lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:37:55 +03:00
Aliaksandr Valialkin
a78b3dba7f app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count 2019-06-19 18:36:47 +03:00
Aliaksandr Valialkin
d2c801029b lib/storage: persist metric ids for the current and the previous hour on graceful shutdown
This should improve performance after restart when the db contains a lot of time series
with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)
2019-06-14 07:55:14 +03:00
Aliaksandr Valialkin
419197ba08 lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func
The func syncs parent dir in order to persist directory removal
in the event of power loss
2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin
935bfd7a18 lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error 2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin
20fc0e0e54 lib/{storage,mergeset}: sync filenames inside part when finalizing the part
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin
ac7b186f13 all: try hard removing directory with contents
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin
cbe692f0e2 app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values 2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin
7b6623558f lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter
This should improve speed for searching metrics among high number of time series
with high churn rate like in big Kubernetes clusters with frequent deployments.
2019-06-10 14:07:39 +03:00
Aliaksandr Valialkin
a1351bbaee lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters 2019-06-10 13:26:44 +03:00
Aliaksandr Valialkin
b4d707d9bb lib/storage: give clearer names to more functions 2019-06-10 13:01:23 +03:00
Aliaksandr Valialkin
bee7298f81 lib/storage: give more clear names to functions 2019-06-10 12:50:44 +03:00
Aliaksandr Valialkin
dbd217b8f0 lib/storage: test GetSeriesCount 2019-06-10 12:43:34 +03:00
Aliaksandr Valialkin
4d936b1524 lib/storage: make getSeriesCount func indexSearch method 2019-06-10 12:29:11 +03:00
Aliaksandr Valialkin
d37924900b lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes) 2019-06-09 19:13:56 +03:00
Aliaksandr Valialkin
28f6c36ab4 lib/storage: tune updating a map with today`s metric ids
- Increase update iterval from 1s to 10s. This should reduce CPU usage
  for large amounts of metric ids with constant churn.
- Reduce pendingTodayMetricIDsLock lock duration during the update.
2019-06-02 21:58:16 +03:00
Aliaksandr Valialkin
4794f894a4 lib/storage: speed up checking metricID existence in the list for the current date 2019-06-02 18:34:08 +03:00
Aliaksandr Valialkin
e307a4d92c lib/timerpool: use timer pool in concurrency limiters
This should reduce the number of memory allocations in highly loaded system
2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin
54fb8b21f9 all: fix misspellings 2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
d6523ffe90 Makefile: add -s flag to go fmt in make fmt command 2019-05-25 21:43:35 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00