Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 20:37:12 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
494 Commits 153 Branches 748 Tags 386 MiB
1e46961d68
Commit Graph

162 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
1e46961d68 app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
 2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
543dc8d337 lib/storage: populate partition names from both small and big directories 
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
 2019-11-06 19:49:34 +02:00
Aliaksandr Valialkin
e472f0b23b lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter 
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.

Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
 2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin
c51ca04a43 lib/storage: take into account the requested time range when caching TSIDs for the given tag filters 2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin
e37f06dc52 lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting 2019-11-05 18:44:22 +02:00
Aliaksandr Valialkin
5c2099ecfe lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions 2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin
885ba17905 lib/storage: separate the max inverted index scan loops per metric into fast and slow loops 
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
 2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin
b9a06e8e74 lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive 
This should improve performance for query filters over big number of time series.
 2019-11-05 14:19:13 +02:00
Aliaksandr Valialkin
30c8301b11 lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name 
The new value reduces the amount of wasted work during index scans over big number of time series.
 2019-11-05 14:19:06 +02:00
Aliaksandr Valialkin
e53f9e553d lib/storage: try potentially faster tag filters at first, then apply slower tag filters 
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
 2019-11-05 14:19:01 +02:00
Aliaksandr Valialkin
3c90d77858 lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method 
This fixes lint errors
 2019-11-04 01:07:19 +02:00
Artem Navoiev
478767d0ed add unittests for bytesutil and storage (#221) 2019-11-04 00:54:46 +02:00
Aliaksandr Valialkin
02e0b19a62 lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
6be4456d88 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
9becc26f4b lib/storage: remove interface conversion in hot path during block merging 
This should improve merge speed a bit for parts with big number of small blocks.
 2019-11-03 12:33:34 +02:00
Aliaksandr Valialkin
c62399eb3e lib/{storage,mergeset}: create missing partition directories after restoring from backups 
Backup tools could skip empty directories. So re-create such directories on the first run.
 2019-11-02 02:27:11 +02:00
Aliaksandr Valialkin
55d728c849 lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones 
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
 2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
808fc0971f lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones 
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
 2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
370cfbb365 lib/uint64set: return an emptry set instead of nil set from Set.Clone, since the caller may add data to the cloned set 
This fixes the following panic in v1.28.1:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e]

goroutine 1155 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)
 2019-11-01 16:12:44 +02:00
Aliaksandr Valialkin
d18ea0c95b app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges 2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin
e0b292c6de lib/storage: small cleanup in Storage.add 2019-10-31 14:30:34 +02:00
Aliaksandr Valialkin
e76e21e4c7 lib/decimal: speed up FromFloat for common case with integers 2019-10-31 13:24:59 +02:00
Aliaksandr Valialkin
cfa5e279c2 lib/decimal: increase float64->decimal conversion precision a bit 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
 2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
26d570bb3a lib/storage: get parts to merge after applying the limit on the number of concurrent merges 
This should reduce write amplification under high ingestion rate.
 2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
2e2eff90d5 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:28 +02:00
Aliaksandr Valialkin
04e48ef064 lib/fs: typo fix in comment to WriteFileAtomically 2019-10-29 11:31:26 +02:00
Aliaksandr Valialkin
a42b5db39f lib/decimal: increase float->decimal conversion precision for big numbers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
 2019-10-28 13:23:44 +02:00
hanzai
b3c946e35a warns during rows addition (#214) 2019-10-20 23:41:07 +03:00
Aliaksandr Valialkin
e83fe938c8 all: make fmt 2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin
97ce4e03a5 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
6ebf537153 lib/memory: properly handle int overflow in sysTotalMemory 
This should fix builds on 32-bit architectures such as arm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 00:50:48 +03:00
Aliaksandr Valialkin
195cefd81a lib/prompb: removed outdated README.md 2019-10-14 22:12:57 +03:00
Aliaksandr Valialkin
f6334bffa1 lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order 2019-10-09 12:13:17 +03:00
Aliaksandr Valialkin
2abd5154e0 lib/storage: typo fix in comment to maxRowsPerSmallPart. 2019-10-08 18:51:20 +03:00
Aliaksandr Valialkin
c1cf7d9f93 lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. 
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
 2019-10-08 16:27:35 +03:00
Aliaksandr Valialkin
fdc5cfd838 lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
 2019-09-30 12:25:40 +03:00
Aliaksandr Valialkin
0fa6df94a2 lib/storage: optimize TSID comparison 2019-09-26 14:16:02 +03:00
Aliaksandr Valialkin
c39355921e lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:41 +03:00
Aliaksandr Valialkin
0db9fcedd5 lib/storage: properly match labels against regexp with (?i) flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-09-26 11:03:10 +03:00
Aliaksandr Valialkin
2444433d83 lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:43 +03:00
Aliaksandr Valialkin
ea4c828bae lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:55:13 +03:00
Aliaksandr Valialkin
aebc45ad26 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 14:02:15 +03:00
Aliaksandr Valialkin
2cb811b42f lib/uint64set: optimize Set.AppendTo 2019-09-25 00:34:17 +03:00
Aliaksandr Valialkin
b986516fbe lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:17:55 +03:00
Aliaksandr Valialkin
ef2296e420 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:34 +03:00
Aliaksandr Valialkin
a6086cde78 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:49:51 +03:00
Aliaksandr Valialkin
c9063ece66 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:35:15 +03:00
Aliaksandr Valialkin
4e26ad869b lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:43:13 +03:00
Aliaksandr Valialkin
0772191975 lib/mergeset: detect whether we are in test by executable suffix 2019-09-22 23:12:15 +03:00
Aliaksandr Valialkin
48999e5396 lib/workingsetcache: remove data race when resetting c.misses 2019-09-22 19:36:49 +03:00