Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-19 07:01:02 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
9,058 Commits 114 Branches 748 Tags 596 MiB
246c339e3d
Commit Graph

9058 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aliaksandr Valialkin
56dff57f77 app/vmselect/netstorage: reduce memory usage when fetching big number of data blocks from vmstorage 
Dump data blocks directly to temporary file instead of buffering them in RAM
 2019-09-28 12:21:57 +03:00
Aliaksandr Valialkin
ba460f62e6 app/vmselect/promql: do not generate timestamps for NaN values in timestamp function according to Prometheus logic 2019-09-27 18:55:16 +03:00
Stian Øvrevåge
a9dac3829e Update README.md - Fix helm command typos 
`victoria-mertrics` -> `victoria-metrics` in helm commands.
 2019-09-27 12:33:48 +03:00
Aliaksandr Valialkin
de919574a5 deployment/docker: switch Go builder image from v1.13.0 to v1.13.1 2019-09-26 17:12:52 +03:00
Aliaksandr Valialkin
d0b4590099 lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f lib/storage: properly match labels against regexp with (?i) flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
bd1cf053f6 app/vmselect/promql: add increases_over_time and decreases_over_time functions 
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
 2019-09-25 20:38:51 +03:00
Aliaksandr Valialkin
4e3871ac1e lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966 lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
8d398af92f app/vminsert/netstorage: mention the data size that cannot be sent to vmstorage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
 2019-09-25 12:53:41 +03:00
Aliaksandr Valialkin
73ac7b8dd6 app/vminsert/netstorage: make sure the conn exists before closing it in storageNode.closeBrokenConn 
The conn can be missing or already closed during the call to storageNode.closeBrokenConn.
Prevent `nil pointer dereference` panic by verifying whether the conn is already closed.

Thanks to @CH-anhngo for reporting the issue.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
 2019-09-25 10:36:50 +03:00
Aliaksandr Valialkin
c64fb91a43 lib/uint64set: optimize Set.AppendTo 2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
3304dc1e85 lib/mergeset: detect whether we are in test by executable suffix 2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin
d2ed8cb0b2 lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID 
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
 2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
0a9cb6368e lib/workingsetcache: remove data race when resetting c.misses 2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin
7d13c31566 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values 
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
 2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
272e2f77c9 lib/encoding: optimize UnmarshalUint* and UnmarshalInt* 2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
7e0c6d4ca6 lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries 
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
 2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
b0c738ae8b app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls 2019-09-19 22:16:11 +03:00
Aliaksandr Valialkin
bf8505353a lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts 2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin
ebbef20535 lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations 2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin
89234f395d lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs 2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects 2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
410f993bf6 lib/mergeset: fill partHeader.firstItem on first block flush 2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin
c05885fb5f lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless 
This will save CPU on subsequent calls for this filter
 2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
e041a196a7 deployment/docker/docker-compose.yml: update Prometheus from v2.3.2 to v2.12.0 2019-09-18 18:30:02 +03:00
Aliaksandr Valialkin
db71c940ea lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there 2019-09-17 23:18:37 +03:00
Artem Navoiev
ccb6dc6925 [deployment] clean up helm (#185) 2019-09-16 21:58:16 +03:00
Aliaksandr Valialkin
491b1317f4 vendor: update github.com/valyala/gozstd from v1.6.1 to v1.6.2 2019-09-16 21:50:02 +03:00
Aliaksandr Valialkin
5666112de2 deployment: switch docker image tag from v1.27.2-cluster to v1.27.3-cluster 2019-09-14 11:33:18 +03:00
Aliaksandr Valialkin
ba21622b78 vendor: make vendor-update 2019-09-13 22:49:34 +03:00
Aliaksandr Valialkin
020341d13a deployment/docker: remove file system paths from the compiled binary 2019-09-13 22:46:07 +03:00
Aliaksandr Valialkin
550a12415a app/vminsert/netstorage: log network errors when sending data to vmstorage nodes 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
 2019-09-13 22:26:24 +03:00
Aliaksandr Valialkin
41ef6b060e lib/mergeset: properly check for sorted block headers 
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
 2019-09-13 21:59:38 +03:00
Aliaksandr Valialkin
ee4585db33 app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step])) 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
 2019-09-13 21:42:11 +03:00
hanzai
08cde5e3f4 lib/workingsetcache: adjust switching from mode=split to mode=whole smoothly and load cachefile successfully 2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin
828e5f6d26 app/vmselect/promql: binary operation fixes according to Prometheus behaviour 
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
 2019-09-13 17:43:09 +03:00
Artem Navoiev
62b424bc4c [ci] github actions - run pipeline on pull request. Fix running of test in external PR from forks 2019-09-11 14:54:45 +03:00
Aliaksandr Valialkin
ed50b8792b app/vminsert/netstorage: reduce the maximum buffer size for rerouted rows, so it occupies less RAM 2019-09-11 14:50:30 +03:00
Aliaksandr Valialkin
b101064f8b all: report the number of bytes read on io.ReadFull error 
This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
 2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin
2f4c950fe9 app/vminsert/netstorage: send per-storageNode bufs to vmstorage nodes in parallel 
This should improve the maximum ingestion throughput.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
 2019-09-11 14:50:19 +03:00
Aliaksandr Valialkin
694cc59ed1 app/vminsert/netstorage: dynamically adjust timeouts for sending packets from vminsert to vmstorage depending on packet size 
Bigger packets will have more chances to be sent to vmstorage.
 2019-09-11 14:50:14 +03:00
Aliaksandr Valialkin
568ff61dcf lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-09 11:42:45 +03:00