Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-19 23:16:26 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
7,084 Commits 114 Branches 748 Tags 607 MiB
c07909a20b
Commit Graph

2107 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
e75d5f47c4 lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs 2019-07-12 02:28:23 +03:00
Aliaksandr Valialkin
fc90ebf43c lib/storage: do not reduce maxMetrics on time ranges exceeding maxDaysForDateMetricIDs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95
 2019-07-12 02:20:34 +03:00
Aliaksandr Valialkin
2bd1a01d1a lib/storage: do not pollute inverted index with data for samples outside the retention period 2019-07-11 17:04:56 +03:00
Aliaksandr Valialkin
d031e04023 lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value 2019-07-11 14:48:37 +03:00
Aliaksandr Valialkin
43ea4ce428 lib/storage: remember and skip individual tag filters matching too many metrics 
This saves CPU time by skipping useless matching for individual tag filters.
 2019-07-11 14:48:30 +03:00
Aliaksandr Valialkin
1fe6d784d8 all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin
56c154f45b all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size 2019-07-04 19:42:30 +03:00
Aliaksandr Valialkin
c7034fc51b lib/memory: attempt #3 to determine memory limit for LXC container 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
 2019-07-01 14:01:13 +03:00
Aliaksandr Valialkin
a41955863a lib/mergeset: make fmt 2019-06-29 14:25:26 +03:00
Aliaksandr Valialkin
2ecb117082 lib/storage: skip non-matching metricIDs in sortedFilter 
This should improve performance for big sorteFilter lists.
 2019-06-29 13:48:32 +03:00
Aliaksandr Valialkin
0c88afa386 lib/mergeset: speed up binarySearchKey by skipping the first item during binary search 2019-06-29 13:45:49 +03:00
Aliaksandr Valialkin
828078eb45 lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis 2019-06-28 18:22:46 +03:00
Aliaksandr Valialkin
7b59466667 lib/memory: attempt #2 to determine memory limit inside LXC container 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
 2019-06-28 18:08:15 +03:00
Aliaksandr Valialkin
593bd35aaa lib/memory: an attempt to read proper memory limit inside LXC container 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
 2019-06-28 15:34:30 +03:00
Aliaksandr Valialkin
c1be1e4342 lib/storage: optimize time series search by regexp filter 
This should improve search speed on label filters like `{foo=~"bar.+baz"}`
 2019-06-27 16:17:43 +03:00
Aliaksandr Valialkin
683bf2a11f lib/storage: make sure non-nil args are passed to openIndexDB 2019-06-25 20:10:04 +03:00
Aliaksandr Valialkin
eb2283a029 lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive 
This should improve performance on inverted index search for big amount of unique time series
when big -search.maxUniqueTimeseries is set.
 2019-06-25 19:55:27 +03:00
Aliaksandr Valialkin
e8377011ab lib/storage: free up memory from caches owned by indexDB when it is deleted 2019-06-25 14:42:44 +03:00
Aliaksandr Valialkin
33ea2120c3 lib/storage: use unversioned keys for tag cache in extDB 
Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache.
This should improve performance for index lookups over big amount of time series.
 2019-06-25 13:08:58 +03:00
Aliaksandr Valialkin
cf63669303 lib/storage: skip searching in extDB if it doesn't contain items for the given time range 
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
 2019-06-25 13:00:37 +03:00
Aliaksandr Valialkin
af2ceaaa0b lib/storage: mention source parts on merge error 
This should improve determining broken source part.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
 2019-06-24 14:08:43 +03:00
Aliaksandr Valialkin
e88a03323a all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 2019-06-20 20:07:10 +03:00
Aliaksandr Valialkin
80db24386e lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:37:55 +03:00
Aliaksandr Valialkin
296c14317f lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:36:15 +03:00
Aliaksandr Valialkin
a78b3dba7f app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count 2019-06-19 18:36:47 +03:00
Aliaksandr Valialkin
710d6c33ea lib/prompb: remove superflouos bytes copying in ReadSnappy 2019-06-18 20:37:51 +03:00
Aliaksandr Valialkin
6d6c9eb1f8 lib/flagutil: remove unused package 2019-06-18 10:43:55 +03:00
Aliaksandr Valialkin
d2c801029b lib/storage: persist metric ids for the current and the previous hour on graceful shutdown 
This should improve performance after restart when the db contains a lot of time series
with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)
 2019-06-14 07:55:14 +03:00
Aliaksandr Valialkin
26f8d7ea1b lib/fs: sync parent dir in MustRemoveAll only if it exists 
The parent directory may be non-existing when the deleted directory
didn't exist before the MustRemoveAll call
 2019-06-12 02:14:44 +03:00
Aliaksandr Valialkin
419197ba08 lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func 
The func syncs parent dir in order to persist directory removal
in the event of power loss
 2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin
c1276edab5 lib/fs: panic with fatal error when directories cannot be removed 
Unremoved directories may lead to inconsistent data directory,
so VictoriaMetrics will fail to start next time.

So panic on the first error when trying to remove directory in order
to simplify recover process.
 2019-06-12 01:20:54 +03:00
Aliaksandr Valialkin
2322c9a45a lib/fs: attempt #2 to work around NFS issue with directory removal 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
 2019-06-12 01:07:05 +03:00
Aliaksandr Valialkin
935bfd7a18 lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error 2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin
3dd36b8088 lib/fs: make sure the created directory remains visible in the fs in the event of power loss 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
 2019-06-11 23:08:09 +03:00
Aliaksandr Valialkin
afb964670a lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused 2019-06-11 22:54:26 +03:00
Aliaksandr Valialkin
20fc0e0e54 lib/{storage,mergeset}: sync filenames inside part when finalizing the part 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
 2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin
ac7b186f13 all: try hard removing directory with contents 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
 2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin
cbe692f0e2 app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values 2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin
7b6623558f lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter 
This should improve speed for searching metrics among high number of time series
with high churn rate like in big Kubernetes clusters with frequent deployments.
 2019-06-10 14:07:39 +03:00
Aliaksandr Valialkin
a1351bbaee lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters 2019-06-10 13:26:44 +03:00
Aliaksandr Valialkin
b4d707d9bb lib/storage: give clearer names to more functions 2019-06-10 13:01:23 +03:00
Aliaksandr Valialkin
bee7298f81 lib/storage: give more clear names to functions 2019-06-10 12:50:44 +03:00
Aliaksandr Valialkin
dbd217b8f0 lib/storage: test GetSeriesCount 2019-06-10 12:43:34 +03:00
Aliaksandr Valialkin
4d936b1524 lib/storage: make getSeriesCount func indexSearch method 2019-06-10 12:29:11 +03:00
Aliaksandr Valialkin
d37924900b lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes) 2019-06-09 19:13:56 +03:00
Aliaksandr Valialkin
a3e26de45e lib/procutil: typo fix in comment to WaitForSigterm 2019-06-04 17:31:47 +03:00
Aliaksandr Valialkin
28f6c36ab4 lib/storage: tune updating a map with today`s metric ids 
- Increase update iterval from 1s to 10s. This should reduce CPU usage
  for large amounts of metric ids with constant churn.
- Reduce pendingTodayMetricIDsLock lock duration during the update.
 2019-06-02 21:58:16 +03:00
Aliaksandr Valialkin
4794f894a4 lib/storage: speed up checking metricID existence in the list for the current date 2019-06-02 18:34:08 +03:00
Aliaksandr Valialkin
e307a4d92c lib/timerpool: use timer pool in concurrency limiters 
This should reduce the number of memory allocations in highly loaded system
 2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin
54fb8b21f9 all: fix misspellings 2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
d6523ffe90 Makefile: add -s flag to go fmt in make fmt command 2019-05-25 21:43:35 +03:00
Aliaksandr Valialkin
6b0ae332f8 lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls 2019-05-24 13:01:02 +03:00
Aliaksandr Valialkin
6fb9dd09f5 lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio 2019-05-24 12:34:32 +03:00
Aliaksandr Valialkin
19b6643e5c lib/encoding: substitute CompressZSTD with CompressZSTDLevel 2019-05-24 12:32:55 +03:00
Aliaksandr Valialkin
08b889ef09 lib/httpserver: add -http.disableResponseCompression flag, which may help saving CPU resources at the cost of higher network bandwidth usage 2019-05-24 12:18:40 +03:00
Aliaksandr Valialkin
674888fdc9 lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29 2019-05-23 17:18:35 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00