Aliaksandr Valialkin
da98703748
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
c28876172f
app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0
2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin
50ae1879c6
app/vmselect/promql: add histogram
aggregate function, which is useful for building heatmaps from multiple time series
2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin
8582b50360
app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets
2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin
19dfe52254
app/vmselect/promql: properly handle histogram_quantile(0, ...)
with zero buckets
2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin
4bb88843cf
app/vmselect: add vm_per_query_{rows,series}_processed_count
histograms
2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
7753c8c0a1
app/vmselect/promql: transparently apply prometheus_buckets
in histogram_quantile
2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin
c4287b3c86
app/vmselect/promql: add prometheus_buckets
function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics
to Prometheus-compatible buckets
2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin
1f3fd2c910
app/vmselect: adjust end
arg instead of adjusting start
arg if start > end
...
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin
7a4635f853
all: remove the remaining mentions of cluster version
2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
443189fb0a
app/{vmbackup,vmrestore}: add -maxBytesPerSecond
command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin
0094bc4fc9
app/vmselect/prometheus: properly adjust too big time time
on /api/v1/query
...
Too big `time` must be adjusted to `now()-queryOffset`.
2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin
3f1637fae8
app/vmselect/promql: properly calculate integrate(q[d])
2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin
c56b9ed03b
app/victoria-metrics: add build rules for GOARCH=ppc64le
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
2019-11-13 20:24:33 +02:00
Aliaksandr Valialkin
3fd32e331a
app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
2019-11-13 20:24:26 +02:00
Aliaksandr Valialkin
119dfd01bb
lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"}
metric
2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin
86a1cd700b
lib/storage: remove inmemory index for recent hour, since it uses too much memory
...
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.
Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin
c57eb0ff83
lib/storage: add -disableRecentHourIndex
flag for disabling inmemory index for recent hour
...
This may be useful for saving RAM on high number of time series aka high cardinality
2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin
ca259864e2
lib/storage: return back inmemory inverted index for recent hour
...
Issues fixed:
- Slow startup times. Now the index is loaded from cache during start.
- High memory usage related to superflouos index copies every 10 seconds.
2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin
01bb3c06c7
lib/storage: remove inmemory inverted index for recent hours
...
Production load with >10M active time series showed it could
slow down VictoriaMetrics startup times and could eat
all the memory leading to OOM.
Remove inmemory inverted index for recent hours until thorough
testing on production data shows it works OK.
2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin
6c2303764e
Revert "lib/fs: do not postpone directory removal on NFS error"
...
This reverts commit 4c02e496f7
.
Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234
2019-11-12 16:18:09 +02:00
Aliaksandr Valialkin
661dd190bb
Refer to https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883 from multiple places in README.md
2019-11-12 13:02:39 +02:00
Oleg Kovalov
b4f44befa3
fix misspelled words ( #229 )
2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
8e8f98f712
lib/storage: add tests for dateMetricIDCache
2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin
56d7cc8a0d
app/victoria-metrics: remove deprecated fs.MustStopDirRemover from main_test.go
2019-11-10 13:37:13 +02:00
Aliaksandr Valialkin
4c02e496f7
lib/fs: do not postpone directory removal on NFS error
...
Continue trying to remove NFS directory on temporary errors for up to a minute.
The previous async removal process breaks in the following case during VictoriaMetrics start
- VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them.
- The transaction instructs removing old directories for parts, which were already merged into bigger part.
- VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors.
- VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished
and finds directories, which should be removed, but weren't still removed due to NFS errors.
- VictoriaMetrics panics when it finds unexpected empty directory.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin
5c3fa59181
app/vmrestore: the upcoming release would be 1.29.0
2019-11-10 00:20:41 +02:00
Aliaksandr Valialkin
ee7765b10d
lib/storage: implement per-day inverted index
2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin
5810ba57c2
lib/storage: use specialized cache for (date, metricID) entries
...
This improves ingestion performance.
2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin
6ad7fe8eeb
lib/storage: export vm_new_timeseries_created_total
metric for determining time series churn rate
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
63b05c0b9f
app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions
...
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.
Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.
This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
d888b21657
lib/storage: add inmemory inverted index for the last hour
...
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
1e46961d68
app/{vmbackup,vmrestore}: add vmbackup
and vmrestore
tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
e472f0b23b
lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter
...
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.
Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin
2f58f37f07
app/vmselect/promql: add lag(q[d])
function, which returns the lag between the current timestamp and the timstamp for the last data point in q
2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin
d18ea0c95b
app/vmstorage: add -bigMergeConcurrency
and -smallMergeConcurrency
flags for tuning the maximum number of CPU cores used during merges
2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin
b0295dbf2e
app/vmselect: add -search.latencyOffset
flag for tuning the time after data collection when data points become visible in query results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin
e83fe938c8
all: make fmt
2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin
97ce4e03a5
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
f752479cb8
app/victoria-metrics/test: add missing docs to public funcs PopulateTimeTplString and PopulateTimeTpl
2019-10-17 00:50:46 +03:00
Aliaksandr Valialkin
61e956e175
app/victoria-metrics: add a test for max_lookback=<duration>
query arg
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
c66a691593
app/vmselect/prometheus: add -search.maxLookback
command-line flag for overriding dynamic calculations for max lookback interval
...
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.
The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
cc21b31502
app/victoria-metrics/test: add a test for PopulateTimeTplString
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
956fdd89d3
app/vmselect/promql: take into account the previous point when calculating max_over_time
and min_over_time
...
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
2019-10-08 12:30:05 +03:00
Artem Navoiev
1e2c511747
Add regression test for query apo
...
Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187
cover:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-10-07 22:18:04 +03:00
Artem Navoiev
a116f5e7c1
Add regression test for query apo ( #194 )
...
Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187
cover:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-30 11:25:54 +03:00
Aliaksandr Valialkin
4e9e1ca0f7
app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially
...
This became the case after b7ee2e7af2
.
2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin
c1d3705be0
app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock
...
This allows re-using the destination buffer for marshaling in the outer loop.
2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin
b7ee2e7af2
app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series
2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin
67d44b0845
app/vmselect/promql: do not generate timestamps for NaN values in timestamp
function according to Prometheus logic
2019-09-27 18:54:43 +03:00
Artem Navoiev
1e6ae9eff4
Add regression test for duplicated labels and series
...
Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187
cover:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/155
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
2019-09-27 16:52:16 +03:00
Artem Navoiev
cf4786f34a
add test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 12:45:19 +03:00
Aliaksandr Valialkin
26dc21cf64
app/vmselect/promql: add increases_over_time
and decreases_over_time
functions
...
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin
d32f88c378
app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls
2019-09-19 22:15:59 +03:00
Aliaksandr Valialkin
d4218d27e6
app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step]))
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-13 21:41:04 +03:00
Aliaksandr Valialkin
4d636c244d
app/vmselect/promql: binary operation fixes according to Prometheus behaviour
...
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
2019-09-13 17:42:52 +03:00
Artem Navoiev
4fd741f40d
[tests] check timestamp in tests ( #177 )
2019-09-08 19:48:38 +03:00
Artem Navoiev
22e3fabefd
Add OpenTSDB and Prometheus integration tests ( #168 )
...
* [WIP] open tsdb and prometheus integration tests
* app/victoria-metrics: fix race condition on parallel tests
2019-09-05 17:55:38 +03:00
Aliaksandr Valialkin
88f8670ede
lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown
...
This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin
6954e126fc
app/vmselect/promql: ignore grouping by destination label in count_values
, since such a grouping is performed automatically
2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin
305e7bc981
app/vmselect/promql: do not return artificial points beyond the last point in time series
2019-09-04 16:35:34 +03:00
Aliaksandr Valialkin
9721d06c6a
app/vmselect/prometheus: do not adjust start
and end
args in /api/v1/query_range
if nocache=1
arg is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171
2019-09-04 13:10:09 +03:00
Aliaksandr Valialkin
db4560ca31
app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does
2019-09-03 20:42:54 +03:00
Aliaksandr Valialkin
1575a560f0
app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks
...
The maximum inmemory file size now depends on `-memory.allowedPercent`.
This should improve performance and reduce the number of filesystem calls
on machines with big amounts of RAM when performing heavy queries
over big number of samples and time series.
2019-09-03 13:32:09 +03:00
Aliaksandr Valialkin
3685fc18d5
Makefile: extract app-local
and app-local-pure
build rules
2019-08-28 01:34:58 +03:00
Aliaksandr Valialkin
ede7ad3703
app/victoria-metrics: add missing victoria-metrics
prefix to --version
output when building with make victoria-metrics
2019-08-28 01:28:08 +03:00
Aliaksandr Valialkin
9196c085a7
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
603a79b357
app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries
2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin
fb909cf710
app/vminsert/influx: set db
label only if Influx line doesnt have db
tag
2019-08-24 13:52:48 +03:00
Aliaksandr Valialkin
9c9221d1b2
app/vminsert: skip empty tags
2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
70ca018a57
app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows
...
Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric
2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
4266091e4f
app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows
...
Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric
2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
8001d29b6e
app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows
...
Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric
2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
9d3f1fcbb9
app/vminsert/influx: skip invalid rows and continue parsing the remaining rows
...
Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.
2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
ba7b3806be
app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life
...
The prefious report with escaped newline chars in influx line protocol was false alarm.
2019-08-23 18:42:05 +03:00
Aliaksandr Valialkin
7fa88c6efc
app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life
2019-08-23 18:35:41 +03:00
Aliaksandr Valialkin
4da34b11f8
app/vminsert/influx: handle \r\n
aka crlf
influx line endings from windows world
...
Such lines exist in real life.
2019-08-23 18:28:49 +03:00
Aliaksandr Valialkin
a18317adbc
app/vminsert/influx: allow escaping newline char
...
Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ ,
there are reports that such chars occur in real life
2019-08-23 15:14:46 +03:00
Aliaksandr Valialkin
44d7fc599d
app/vminsert/influx: skip comments starting with #
in influx line protocol
2019-08-23 14:43:09 +03:00
Aliaksandr Valialkin
ac004665b5
all: return 503 http error if service is temporarily unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin
8c03a8c4b4
app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries
2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin
1272e407b2
app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
2019-08-22 14:39:46 +03:00
Aliaksandr Valialkin
5f33fc8e46
app/vminsert: add ability to ingest data via HTTP OpenTSDB /api/put
requests
...
This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152
Thanks to nustinov@gmail.com for the initial pull request.
2019-08-22 12:28:32 +03:00
Aliaksandr Valialkin
ec8125606d
app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing put
prefixes to each line
2019-08-21 19:14:47 +03:00
Aliaksandr Valialkin
f4a38f7fb1
app/vmselect/promql: fix panic on -search.disableCache
...
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin
ab740afd0d
app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue
2019-08-21 11:29:24 +03:00
Aliaksandr Valialkin
7b5168adfb
app/vmselect/promql: remove NaNs from /api/v1/query_range
output like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
2019-08-20 23:01:41 +03:00
Aliaksandr Valialkin
a0d480fbf3
app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries
...
This should reduce memory allocations for big number of timeseries
2019-08-20 23:01:39 +03:00
Aliaksandr Valialkin
d3fd113a80
app/vmselect/promql: add label_value(q, label_name)
func, which returns numeric value labels with name label_name
in q
2019-08-20 00:28:34 +03:00
Aliaksandr Valialkin
dd86e6130c
app/vmselect/promql: independently track offset hints for tStart and tEnd
...
This should improve performance if timeseries starts or ends on the selected time range
2019-08-19 13:40:14 +03:00
Aliaksandr Valialkin
6a27657d73
app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do
...
This should improve the performance of queries over big number of time series
with big number of output points.
2019-08-19 13:03:29 +03:00
Aliaksandr Valialkin
e74fb23189
app/vmselect/promql: add scrape_interval(q[d])
function, which would return scrape interval for q
over d
2019-08-18 21:08:26 +03:00
Aliaksandr Valialkin
582fdc059a
app/vmselect/promql: hande comparisons with NaN
similar to Prometheus
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-08-18 00:25:50 +03:00
Aliaksandr Valialkin
1c108fc494
app/vmselect/promql: add lifetime(q[d])
function, which returns the lifetime of q
over d
in seconds.
...
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
2019-08-16 11:59:32 +03:00
Aliaksandr Valialkin
d6b5ed6d39
app/vmselect/promql: fix corner-case calculation for ideriv
2019-08-16 11:59:28 +03:00
Aliaksandr Valialkin
639b14e8ab
app/vmselect/promql: properly handle corner cases for rollup functions
2019-08-15 23:29:59 +03:00
Aliaksandr Valialkin
9ecb994671
app/vmselect/promql: store compressed results in the cache
...
This should increase rollup results cache capacity.
2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin
09fc6e22e5
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin
b8bb74ffc6
app/vmstorage: add vm_concurrent_addrows_*
metrics for tracking concurrency for Storage.AddRows calls
...
Track also the number of dropped rows due to the exceeded timeout
on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`
2019-08-06 15:08:33 +03:00
Aliaksandr Valialkin
5c83f8e203
app: add vm_concurrent_
metrics for visibility in concurrency limiters for vminsert and vmselect
2019-08-05 18:30:57 +03:00