Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-20 23:46:23 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
8,502 Commits 114 Branches 748 Tags 625 MiB
f69495cd5f
Commit Graph

770 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
f3a7e6f6e3 lib/storage: remove obsolete code 2020-11-02 19:17:30 +02:00
Aliaksandr Valialkin
901514be88 lib/storage: drop more samples outside the given retention during background merge 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
 2020-10-31 20:44:47 +02:00
Aliaksandr Valialkin
7599e5c835 lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag* 2020-10-20 20:09:52 +03:00
Aliaksandr Valialkin
9c5cd5a6c5 lib/storage: code cleanup after 5bfd4e6218 2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin
0db7c2b500 app/vmstorage: support for -retentionPeriod smaller than one month 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
 2020-10-20 14:42:46 +03:00
Aliaksandr Valialkin
efb1989193 lib/storage: small code adjustements after d2960a20e0 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
 2020-10-17 01:17:12 +03:00
faceair
8ddf089deb evaluate the execution cost of all tag filters (#824) 
* evaluate the execution cost of all tag filters

* fix suffixes typo
 2020-10-17 01:13:20 +03:00
Aliaksandr Valialkin
d2e917d1cb app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start 2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin
b51fa16177 app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it 2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin
fd7dd5064a lib/storage: code cleanup after 10f2eedee0 
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.

While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
 2020-10-01 19:12:04 +03:00
Aliaksandr Valialkin
3ad7566a87 lib/storage: imrpove cache effectiveness for time series ids matching the given filters 
Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day.
This should reduce CPU usage in the following cases:

* when querying recently added data with small churn rate for time series
* when querying historical data
 2020-10-01 14:39:46 +03:00
Aliaksandr Valialkin
7c2e4e267a lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space if there are multiple partitions with deferred merges due to disk space shortage 2020-09-29 22:53:34 +03:00
Aliaksandr Valialkin
097a4c10dd app/vmstorage: add metrics for determining whether background merges need additional disk space to complete 
These metrics are:

* vm_small_merge_need_free_disk_space
* vm_big_merge_need_free_disk_space

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin
338a53ccf9 lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm 2020-09-29 13:10:37 +03:00
Aliaksandr Valialkin
ef416c72c2 lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm 2020-09-29 12:42:25 +03:00
Aliaksandr Valialkin
6d8c23fdbd app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native 
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
 2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
533bf76a12 lib/storage: correctly use maxBlockSize in various checks 
Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.
 2020-09-24 18:13:15 +03:00
Aliaksandr Valialkin
31e341371b lib/storage: code prettifying after be5e1222f3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
 2020-09-22 00:42:20 +03:00
faceair
ad41e39350 add filter to getMetricIDs (#783) 
* add getMetricIDs filter

* check nil filter before use
 2020-09-22 00:42:19 +03:00
Aliaksandr Valialkin
a9321f6a60 lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges 2020-09-21 15:51:26 +03:00
Aliaksandr Valialkin
778ea183ca lib/decimal: properly store Inf values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-18 19:08:53 +03:00
Aliaksandr Valialkin
d96858b921 lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions 
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin
3abbb38254 lib/{mergeset,storage}: compare errors with errors.Is() 2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin
ddb3519e17 lib/{mergeset,storage}: code prettifying 2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin
bf826dd828 lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions 2020-09-17 01:49:08 +03:00
Aliaksandr Valialkin
81c05f669b lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin
f307e6f432 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
f5cb213ef9 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin
4ce1368e4b lib/storage: mention time range used in the query that led to error message 
This should improve detecting slow queries with too big time ranges
 2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin
f92255e803 lib/storage: mention tag filters used in the query that led to error message 
This should improve detecting invalid or heavy queries that lead to errors.
 2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin
b3d4ff7ee2 app/vmstorage: improve error logging when the request times out 2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin
307281e922 lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit 
This should improve data ingestion performance when heavy searches are executed

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin
dd1d59f57a lib/storage: properly check timeouts and pace limits 
Previously they were checked on every iteration for small number of iterations
 2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin
a2039b3bbc app/vmselect: return the upper bound on the number of found time series from storage.Search.Init 
This is used by a single-node version in order to reduce memory allocations during search.
See bc8381613d for details.
 2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin
b690eeff53 lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x 2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin
13f8644f8e lib/storage: optimize prefetching metric names for the given metricIDs 2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin
a3e91c593b lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
3149af624d lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 
Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this
increases query latencies since more CPUs are busy with merges.

While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.
 2020-07-31 17:53:13 +03:00
Aliaksandr Valialkin
29bbab0ec9 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
96039dcb40 lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once 
Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series,
while only a single increment is needed when inserting the first data point for this time series.
 2020-07-30 16:17:19 +03:00
Sasasu
96bc476e53 lib/storage: metaindexRow use memroy more efficiently (#655) 
due to memory align the metaindexRow structure use 64-byte pre object.
this commit changes the order of field, make metaindexRow use 56-byte pre
object.

Signed-off-by: Sasasu <su@sasasu.me>
 2020-07-27 23:23:25 +03:00
Aliaksandr Valialkin
94cc677b0c lib/storage: slightly reduce code difference between single-node and cluster versions 2020-07-24 01:18:05 +03:00
Aliaksandr Valialkin
fb3d1380ac lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin
dbf3038637 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin
b8303afcd8 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
7d0743422b lib/storage: properly calculate global metrics in UpdateStats() 2020-07-23 00:35:31 +03:00
Aliaksandr Valialkin
23fa44e56e lib/storage: reorder mergeBlockStreams() args in order to make them more consistent 2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin
754eac676d lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs 
This condition may occur after the following sequence of events:

1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs.
2) All the goroutines return from Storage.AddRows.
3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body.

The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal().
This may take indefinite time.
 2020-07-22 21:52:42 +03:00
Aliaksandr Valialkin
67be79a0bc lib/uint64set: optimize adding items to the set via Set.AddMulti 2020-07-21 20:57:05 +03:00
Aliaksandr Valialkin
be0ab4fbfe lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted 
This is a follow-up commit after 12b16077c4 ,
which didn't reset the `tsidCache` in all the required places.
This could result in indefinite errors like:

    missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time

Fix this by resetting the cache inside deleteMetricIDs function.
 2020-07-14 14:05:19 +03:00
Aliaksandr Valialkin
7335743d57 lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS 
Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense,
since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth
is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs,
i.e. to GOMAXPROCS.
 2020-07-08 17:34:27 +03:00
Aliaksandr Valialkin
fad008df7e lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag 2020-07-08 13:54:13 +03:00
Aliaksandr Valialkin
fe58462bef lib/storage: reset MetricName->TSID cache after deleting time series 
This should prevent from adding new data points to deleted time series
without the need to check for the deleted time series.

This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set
contains big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596

Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604
 2020-07-06 22:01:24 +03:00
Aliaksandr Valialkin
0bff96fe4b lib/storage: prioritize data ingestion over heavy queries 
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.

Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
 2020-07-05 19:44:04 +03:00
Aliaksandr Valialkin
8bb3622e9d app/vminsert: prevent from adding and/or selecting labels with empty values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
 2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin
4cb3e7595c app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
70bf8218bb app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:32:27 +03:00
Tristan Su
c254b683fd lib/storage: set big/small merge concurrency (#568) 
fixed #567

Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>
 2020-06-19 02:21:55 +03:00
Aliaksandr Valialkin
4f673a5201 app/vminsert: export metrics for determining ingested rows with dropped or truncated labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
 2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin
5f3a895c23 lib/storage: add key!=".+" filter additionally to negative filter matching empty value such as key!~"|foo" 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
 2020-06-18 20:05:45 +03:00
Aliaksandr Valialkin
c40f29f783 lib/storage: properly match {tag!="|foo"} filters 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
 2020-06-10 19:34:37 +03:00
Aliaksandr Valialkin
3d0a0b3785 lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds 2020-06-04 13:14:04 +03:00
Aliaksandr Valialkin
eca1afdc20 lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0 2020-05-28 11:58:47 +03:00
Aliaksandr Valialkin
b0131c79b6 lib/storage: improve search speed for time series matching Graphite whildcards such as foo.*.bar.baz 
Add index for reverse Graphite-like metric names with dots. Use this index during search for filters
like `__name__=~"foo\\.[^.]*\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case.

This change may "hide" historical time series during queries. The workaround is to add `[.]*` to the end of regexp label filter,
i.e. "foo\\.[^.]*\\.bar\\.baz" should be substituted with "foo\\.[^.]*\\.bar\\.baz[.]*".
 2020-05-27 21:48:08 +03:00
Aliaksandr Valialkin
2a8f1e6931 lib/storage: do not increment vm_slow_metric_name_loads_total counter for metric_ids which shouldnt be prefetched, since this may mislead users 2020-05-16 10:23:39 +03:00
Aliaksandr Valialkin
1e5c1d7eaa app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance 2020-05-15 14:12:24 +03:00
Aliaksandr Valialkin
d6b9a49481 app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series 2020-05-15 13:46:57 +03:00
Aliaksandr Valialkin
a72f18e821 lib/{storage,mergeset}: further tuning of compression levels depending on block size 
This should improve performance for querying newly added data, since it can be unpacked faster.
 2020-05-15 13:12:28 +03:00
Aliaksandr Valialkin
2cf2e9955b lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish 2020-05-15 12:12:20 +03:00
Aliaksandr Valialkin
67e331ac62 lib/storage: optimize ingestion pefrormance for new time series 2020-05-15 12:12:19 +03:00
Aliaksandr Valialkin
1b5d272e07 lib/storage: reduce indentation in Storage.add 2020-05-14 23:23:56 +03:00
Aliaksandr Valialkin
71d29a8fa1 lib/storage: return the first error instead of the last error, since the first error usually points to the root cause 2020-05-14 23:18:59 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
7e831741f9 lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused 2020-05-14 15:32:23 +03:00
Aliaksandr Valialkin
2f42b85e0e lib/storage: document that getnerateUniqueMetricID should return dense ids 2020-05-14 14:08:59 +03:00
Aliaksandr Valialkin
f442d81648 lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse 2020-05-14 14:03:15 +03:00
Aliaksandr Valialkin
8bb44a5d09 lib/storage: optimize label matching for regexp ending with literal suffix 
For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`,
so it should work faster now.
 2020-05-13 11:39:05 +03:00
Aliaksandr Valialkin
bd5f4e0344 lib/storage: properly initialize part struct before trying to close it on error 
This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .
 2020-05-12 14:54:16 +03:00
Aliaksandr Valialkin
f7753b1469 lib/storage: gradually pre-populate per-day inverted index for the next day 
This should prevent from CPU usage spikes at 00:00 UTC every day when
inverted index for new day must be quickly created for all the active time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430
 2020-05-12 12:13:32 +03:00
Aliaksandr Valialkin
8c77cb436a lib/storage: typo fixes in error messages: or -> of 2020-05-12 12:12:33 +03:00
Aliaksandr Valialkin
bbf06a4248 lib/storage: speed up matching for common regexps in label filters 
The following regexps have been optimized:

* 'foo.+bar'
* 'foo.+bar.+baz'

This should improve performance for matching Graphite-like metrics.
 2020-05-11 22:49:01 +03:00
Aliaksandr Valialkin
37254a139a lib/storage: add a benchmark for Graphite-like regexps for metric names 2020-05-11 22:49:00 +03:00
Aliaksandr Valialkin
d78ed50edd lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown 
Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet.
Log about this and delete the corresponding metricID, so it could be re-created next time.
 2020-04-28 12:01:32 +03:00
Aliaksandr Valialkin
e933cbac16 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin
b16e19c053 lib/storage/dedup.go: go fmt 2020-04-26 14:37:36 +03:00
Aliaksandr Valialkin
a0000c3a6e lib/storage: improve deduplication algorithm 
Now it leaves only the first data point on each `-dedup.minScrapeInterval` interval.

Previously it may leave two data points on the interval. This could lead to unexpected results
for `histogram_quantile(phi, sum(rate(buckets)) by (le))` query.
 2020-04-26 13:10:18 +03:00
Aliaksandr Valialkin
13b4069c59 lib/storage: postpone label filters matching too many time series instead of giving up with error 
This should reduce the frequency of the following errors:

    cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters

    more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range
 2020-04-24 21:18:52 +03:00
Aliaksandr Valialkin
f9526809e5 app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
e9d9638627 lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search 
This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet.

This should fix the following error:

error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF
 2020-04-15 00:10:11 +03:00
Aliaksandr Valialkin
e0c6da8e2a lib/storage: disable deduplication after dedup tests are complete 
The rest of tests expect that the de-duplication is disabled.
 2020-04-10 17:33:38 +03:00
Aliaksandr Valialkin
8ed0d5471a lib/storage: correctly handle -dedup.minScrapeInterval values smaller than 8ms 
Such small values may be used for removing samples with duplicate timestamps.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.
 2020-04-10 16:40:41 +03:00
Aliaksandr Valialkin
0b2f678d8e lib/{storage,mergeset}: make sure that requests and misses cache counters never go down 2020-04-10 14:44:52 +03:00
Aliaksandr Valialkin
0ad7aaf535 lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init 2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin
4c56acbafa lib/storage: remove duplicate data points on 7/8*minScrapeInterval interval instead of 1/2*minScrapeInterval 
This should reduce storage usage and should improve deduplication accuracy
 2020-04-01 15:47:04 +03:00
Aliaksandr Valialkin
504ea876f2 lib/storage: handle errors returned from TagFilters.Add when cloning TagFilters with negative filter 2020-03-31 16:18:34 +03:00
Aliaksandr Valialkin
ef714e01c1 lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet 2020-03-31 12:35:15 +03:00
Aliaksandr Valialkin
7e755b4bac lib/storage: optimize per-day inverted index search for tag filters matching big number of time series 
- Sort tag filters in the ascending number of matching time series
  in order to apply the most specific filters first.
- Fall back to metricName search for filters matching big number of time series
  (usually this are negative filters or regexp filters).
 2020-03-31 00:53:29 +03:00
Aliaksandr Valialkin
d450249955 lib/storage: properly handle {label=~"foo|"} filters as Prometheus does 
Such filters must match all the time series with `label="foo"` plus all the time series without `label`

Previously only time series with `label="foo"` were matched.
 2020-03-30 20:21:47 +03:00
Aliaksandr Valialkin
7cdac6634c lib/storage: serialize snapshot creation process with mutex 
This guarantees that the snapshot contains all the recently added data
from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.
 2020-03-24 22:27:28 +02:00
Aliaksandr Valialkin
31a533656e lib/storage: remove obsolete code 2020-03-13 22:42:42 +02:00
Aliaksandr Valialkin
cf9aee4ec3 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin
110cce24d9 lib/storage: add vm_ prefix to deduplicated_samples_total metric 2020-02-21 19:33:36 +02:00
Aliaksandr Valialkin
a2b81b71b9 lib/storage: typo fix 2020-02-16 15:53:48 +02:00
Aliaksandr Valialkin
ad4cb9f3ca lib/storage: prevent from clobbering nin-nil lastError in Storage.add 2020-02-16 15:51:35 +02:00
Aliaksandr Valialkin
347aaba79d lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate 
It has been appeared that time.Timer was used in places where time.Ticker must be used instead.
This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
 2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin
ea66212c93 lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version 2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin
56d6b8ed0a lib/storage: do not deduplicate blocks with less than 32 samples during merge 
This should improve deduplication accuracy for blocks with higher number of samples.
 2020-02-04 18:41:37 +02:00
Aliaksandr Valialkin
7cde594696 all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
e3adc095bd all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin
a45f25699c lib/storage: re-use indexSearch inside Storage.prefetchMetricNames 2020-01-31 01:18:53 +02:00
Aliaksandr Valialkin
da19fffa08 all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
1332ddc15e lib/storage: pass missing AccountID and ProjectID to searchMetricName 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
4ed5e9a7ce lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init 
This should speed up Search.NextMetricBlock loop for big number of found time series.
 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
ea53a21b02 all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
62b041e90a lib/{mergeset,storage}: properly update lastAccessTime in index and data block cache entries 2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin
a851c75703 lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64) 2020-01-17 23:59:19 +02:00
Aliaksandr Valialkin
476c7fb109 lib/storage: reduce memory allocations when merging metricID sets 2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin
7d429e2806 lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods 
Iterate items with newly added Set.ForEach method instead of allocating `[]uint64`
slice for all the items before the iteration.
 2020-01-15 12:15:48 +02:00
Aliaksandr Valialkin
caffb0cd01 lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm) 2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin
b03ccbf6f7 lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches 
This should reduce memory usage in the long run when old blocks and indexes
aren't accessed anymore.
 2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin
53e176ed67 lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition 
This should reduce write amplification for high ingestion rate on multi-CPU systems
 2020-01-04 23:58:23 +02:00
Aliaksandr Valialkin
a37a006f11 lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores 2019-12-19 18:17:05 +02:00
Aliaksandr Valialkin
8d79412b26 lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series 
This should speed up `/api/v1/import` and make it more scalable on multi-core systems.
 2019-12-19 15:13:36 +02:00
Aliaksandr Valialkin
3694efd005 lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup 
This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.
 2019-12-09 15:45:23 +02:00
Aliaksandr Valialkin
639967db59 lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions call. 
`runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions
can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction
deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction
deletions are finished before returning from `runTransactions`.
 2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin
534da0a8c3 lib/storage: fall back to global inverted index if a filter match too many time series in per-day index 
Previously this resulted to error message. The query may succeed via search in global index.
 2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin
6eb698d1cc lib/storage: fix printing tag filters in TagFilters.String 2019-12-03 14:25:20 +02:00
Aliaksandr Valialkin
c04f60db35 lib/storage: print __name__ instead of empty string in user-visible tag filters 2019-12-03 14:18:18 +02:00
Aliaksandr Valialkin
625f6ca761 lib/storage: optimize regexp filter search 2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin
b9616c017f lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed 
This should fix the issue on NFS when incompletely removed dirs may be left
after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction
files are already removed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin
4e22b521c2 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin
5a62415bec lib/storage: protect from time drift during indexdb rotation 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248
 2019-12-02 14:43:11 +02:00
Aliaksandr Valialkin
f055dbefda lib/storage: generate more human-friendly result in TagFilters.String 2019-12-02 13:56:40 +02:00
Aliaksandr Valialkin
0f184affa7 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
b9e53490b9 lib/storage: move non-matching tag filters to the top at matchTagFilters 
This should reduce the amount of useless work needed for matching the next metricNames.
 2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin
33d9d63393 lib/storage: speed up time series search for queries with multiple filters 
Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.
 2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin
a02a57fbe9 lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers 2019-11-20 15:40:03 +02:00
Aliaksandr Valialkin
6ca4b94511 lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues 
The previous commit was accidentally creating 10x smaller number of time series than Prometheus
and this led to invalid benchmark results.

The updated benchmark results:

benchmark                                                          old ns/op      new ns/op     delta
BenchmarkHeadPostingForMatchers/n="1"                              272756688      6194893       -97.73%
BenchmarkHeadPostingForMatchers/n="1",j="foo"                      138132923      10781372      -92.19%
BenchmarkHeadPostingForMatchers/j="foo",n="1"                      134723762      10632834      -92.11%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"                     195823953      10679975      -94.55%
BenchmarkHeadPostingForMatchers/i=~".*"                            7962582919     100118510     -98.74%
BenchmarkHeadPostingForMatchers/i=~".+"                            7589543864     154955671     -97.96%
BenchmarkHeadPostingForMatchers/i=~""                              1142371741     258003769     -77.42%
BenchmarkHeadPostingForMatchers/i!=""                              9964150263     159783895     -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"              216995884      10937895      -94.96%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"       202541348      10990027      -94.57%
BenchmarkHeadPostingForMatchers/n="1",i!=""                        486285711      87004349      -82.11%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"                350776931      53342793      -84.79%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"              380888565      54256156      -85.76%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"             89500296       21823279      -75.62%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"       379529654      46671359      -87.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"     424563825      53915842      -87.30%

VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)
 2019-11-18 19:48:27 +02:00
Aliaksandr Valialkin
6f61fd367a lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus 
See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52)

The benchmark allows performing apples-to-apples comparison of time series search
in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness -
contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix it.

Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots:

- Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers
- VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers

Benchmark results:
benchmark                                                          old ns/op      new ns/op     delta
BenchmarkHeadPostingForMatchers/n="1"                              272756688      364977        -99.87%
BenchmarkHeadPostingForMatchers/n="1",j="foo"                      138132923      1181636       -99.14%
BenchmarkHeadPostingForMatchers/j="foo",n="1"                      134723762      1141578       -99.15%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"                     195823953      1148056       -99.41%
BenchmarkHeadPostingForMatchers/i=~".*"                            7962582919     8716755       -99.89%
BenchmarkHeadPostingForMatchers/i=~".+"                            7589543864     12096587      -99.84%
BenchmarkHeadPostingForMatchers/i=~""                              1142371741     16164560      -98.59%
BenchmarkHeadPostingForMatchers/i!=""                              9964150263     12230021      -99.88%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"              216995884      1173476       -99.46%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"       202541348      1299743       -99.36%
BenchmarkHeadPostingForMatchers/n="1",i!=""                        486285711      11555193      -97.62%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"                350776931      5607506       -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"              380888565      6380335       -98.32%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"             89500296       2078970       -97.68%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"       379529654      6561368       -98.27%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"     424563825      6757132       -98.41%

The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics.

Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.
 2019-11-18 18:47:02 +02:00
Aliaksandr Valialkin
d297b65089 lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"} metric 2019-11-13 20:26:05 +02:00
Aliaksandr Valialkin
494ad0fdb3 lib/storage: remove inmemory index for recent hour, since it uses too much memory 
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.

Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
 2019-11-13 18:08:58 +02:00
Aliaksandr Valialkin
633dd81bb5 lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour 
This may be useful for saving RAM on high number of time series aka high cardinality
 2019-11-13 15:10:12 +02:00
Aliaksandr Valialkin
f1620ba7c0 lib/storage: fix inmemory inverted index issues found in v1.29 
Issues fixed:
- Slow startup times. Now the index is loaded from cache during start.
- High memory usage related to superflouos index copies every 10 seconds.
 2019-11-13 13:35:38 +02:00
Mike Poindexter
955a592106 Add test for invalid caching of tsids (#232) 
* Add test for invalid caching of tsids

* Clean up error handling
 2019-11-12 15:52:46 +02:00
Oleg Kovalov
74ba42d111 fix misspelled words (#229) 2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin
c48e39eea9 lib/storage: add tests for dateMetricIDCache 2019-11-11 13:21:05 +02:00
Aliaksandr Valialkin
6bdde0d6d4 lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has 2019-11-10 22:04:23 +02:00
Aliaksandr Valialkin
9ea2bd822e lib/storage: implement per-day inverted index 2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin
dea2f3efed lib/storage: use specialized cache for (date, metricID) entries 
This improves ingestion performance.
 2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin
9a43902bd8 lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero 2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin
c16e17dede lib/storage: properly set time range when deleting time series 2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin
8126007c15 lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster 2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin
50773348d3 lib/storage: small code prettifying 2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
0bc54c23ce lib/storage: inmemoryInvertedIndex prettifying 2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
46e67bb78c lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate 2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin
0063c857f5 lib/storage: add inmemory inverted index for the last hour 
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
 2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin
89c03a5464 lib/storage: populate partition names from both small and big directories 
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
 2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin
1c777e0245 lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter 
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.

Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c567a4353a lib/storage: take into account the requested time range when caching TSIDs for the given tag filters 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c6564c5d26 lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting 2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin
a10c4cad85 lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions 2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
e5b1fa0c38 lib/storage: separate the max inverted index scan loops per metric into fast and slow loops 
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
 2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
f93c4f2493 lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive 
This should improve performance for query filters over big number of time series.
 2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin
f48e97263c lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name 
The new value reduces the amount of wasted work during index scans over big number of time series.
 2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin
d2f688c550 lib/storage: try potentially faster tag filters at first, then apply slower tag filters 
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
 2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin
2a38d30f93 lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method 
This fixes lint errors
 2019-11-04 01:06:45 +02:00
Artem Navoiev
e05500cbd4 add unittests for bytesutil and storage (#221) 2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin
f5fbc3ffd7 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
386c349c8c lib/storage: remove interface conversion in hot path during block merging 
This should improve merge speed a bit for parts with big number of small blocks.
 2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin
26ffc77622 lib/{storage,mergeset}: create missing partition directories after restoring from backups 
Backup tools could skip empty directories. So re-create such directories on the first run.
 2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
6ab9c98a1e app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges 2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin
6a22727676 lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups 2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
ca480915ca lib/storage: small cleanup in Storage.add 2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin
a27e034a40 lib/storage: get parts to merge after applying the limit on the number of concurrent merges 
This should reduce write amplification under high ingestion rate.
 2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin
5d2276dbf7 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:37 +02:00
hanzai
52778da1f3 warns during rows addition (#214) 2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin
78fc35c9b1 all: make fmt 2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
661b8ede5b lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order 2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin
7f4a04ee6a lib/storage: typo fix in comment to maxRowsPerSmallPart. 2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin
7e410e1412 lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. 
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
 2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
d0b4590099 lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f lib/storage: properly match labels against regexp with (?i) flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
4e3871ac1e lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966 lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
d2ed8cb0b2 lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID 
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
 2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
7d13c31566 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values 
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
 2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
7e0c6d4ca6 lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries 
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
 2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
89234f395d lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs 2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects 2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
c05885fb5f lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless 
This will save CPU on subsequent calls for this filter
 2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
db71c940ea lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there 2019-09-17 23:18:37 +03:00
Aliaksandr Valialkin
568ff61dcf lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin
2c2bd897dd lib/storage: remove duplicate tag keys on MetricName.Marshal call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
 2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin
0b0153ba3d lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
 2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
604a4312f9 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
da07a6fb38 lib/storage: go fmt 2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin
a63b69e9e2 lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters 2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin
82e813bad3 lib/storage: properly handle (?i) in the tag filter regexp 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin
e2eac858b5 lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159

This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
 2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin
0a8dd9cc9a lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
This should improve query speed over recent data on machines with big amounts of RAM
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
bc576fb386 lib/storage: properly limit the number of output rows in small and big parts storage 
Previously small parts storage didn't take into account the available disk space for big parts.
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
947decb3dd lib/storage: remove outdated comment on maxRowsPerSmallPart 
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
 2019-08-25 13:46:10 +03:00
Aliaksandr Valialkin
e734076f0f app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries 2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin
4ed63d033a lib/storage: add benchmarks for regexp filter match / mismatch 
These benchmarks allow estimate the performance of regexp filters in promql
 2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin
6ec6a8d7c1 lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error 2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin
c59f5c4865 lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders 
This reduces memory usage and memory fragmentation when working with big number of time series
 2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin
99eed2ca14 lib/storage: properly cache tagFilters -> TSIDs entries from historical index 2019-08-14 02:32:25 +03:00
Aliaksandr Valialkin
f1d81b9405 lib/storage: compress contents of cache for tagFilters -> TSIDs 
This should increase cache capacity
 2019-08-14 02:32:22 +03:00
Aliaksandr Valialkin
8c2158af24 all: use workingsetcache instead of fastcache 
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.

The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
 2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin
5a7ab0d90b lib/storage: remove broken BenchmarkIndexDBSearchTSIDs 2019-08-13 20:21:23 +03:00
Aliaksandr Valialkin
39f3f3a517 lib: move common code for creating flock.lock file into fs.CreateFlockFile 2019-08-13 01:46:20 +03:00
Aliaksandr Valialkin
73f866d874 lib/fs: atomically create file with the given contents on WriteFileAtomically 
This should prevent from `transaction` and `metadata.json` files corruption
on unclean shutdown such as OOM, `kill -9`, power loss, etc.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148
 2019-08-12 15:02:04 +03:00
Aliaksandr Valialkin
4fb635b0c9 lib/storage: do not change timestamps to constant rate if values are constant or have constant delta 
This breaks the original timestamps, which results in issues like
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .
 2019-08-06 15:40:17 +03:00
Aliaksandr Valialkin
f56c1298ad app/vmstorage: add vm_concurrent_addrows_* metrics for tracking concurrency for Storage.AddRows calls 
Track also the number of dropped rows due to the exceeded timeout
on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`
 2019-08-06 15:08:43 +03:00
Aliaksandr Valialkin
a3ecf3c1f7 lib/storage: properly reset partSearch.fetchData in partSearch.reset 2019-08-05 09:55:50 +03:00
Aliaksandr Valialkin
880b1d80b1 app/vmselect: optimize /api/v1/series by skipping storage data 
Fetch and process only time series metainfo.
 2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
b7c4b0c6d2 lib/storage: fix matching against tag filter with empty name 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137
 2019-07-30 15:15:21 +03:00
Aliaksandr Valialkin
c6bec48927 lib/storage: add metrics for calculating skipped rows outside the retention 
The metrics are:

    - vm_too_big_timestamp_rows_total
    - vm_too_small_timestamp_rows_total
 2019-07-26 14:11:56 +03:00
Aliaksandr Valialkin
73a47d2a53 lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs 2019-07-12 02:28:40 +03:00
Aliaksandr Valialkin
97f9397687 lib/storage: do not reduce maxMetrics on time ranges exceeding maxDaysForDateMetricIDs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95
 2019-07-12 02:21:52 +03:00
Aliaksandr Valialkin
4ca66344ee lib/storage: do not pollute inverted index with data for samples outside the retention period 2019-07-11 17:11:33 +03:00
Aliaksandr Valialkin
0522efb2d6 lib/storage: add missing tagFilter.Marshal func 2019-07-11 15:01:01 +03:00
Aliaksandr Valialkin
12b1d67b41 lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value 2019-07-11 14:48:51 +03:00
Aliaksandr Valialkin
bf2e1b0ac1 lib/storage: remember and skip individual tag filters matching too many metrics 
This saves CPU time by skipping useless matching for individual tag filters.
 2019-07-11 14:48:47 +03:00
Aliaksandr Valialkin
ba8195c58e all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
41f512af1c all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size 2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin
ffc1bb00f6 lib/storage: skip non-matching metricIDs in sortedFilter 
This should improve performance for big sorteFilter lists.
 2019-06-29 13:49:40 +03:00
Aliaksandr Valialkin
416d27ef11 lib/storage: optimize time series search by regexp filter 
This should improve search speed on label filters like `{foo=~"bar.+baz"}`
 2019-06-27 16:18:00 +03:00
Aliaksandr Valialkin
ee23a143b9 lib/storage: make sure non-nil args are passed to openIndexDB 2019-06-25 20:10:08 +03:00
Aliaksandr Valialkin
8b0a63722f lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive 
This should improve performance on inverted index search for big amount of unique time series
when big -search.maxUniqueTimeseries is set.
 2019-06-25 19:57:31 +03:00
Aliaksandr Valialkin
0263cb0adc lib/storage: free up memory from caches owned by indexDB when it is deleted 2019-06-25 14:41:16 +03:00
Aliaksandr Valialkin
362e187011 lib/storage: use unversioned keys for tag cache in extDB 
Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache.
This should improve performance for index lookups over big amount of time series.
 2019-06-25 13:15:42 +03:00
Aliaksandr Valialkin
51e2f3b48f lib/storage: skip searching in extDB if it doesn't contain items for the given time range 
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
 2019-06-25 12:57:56 +03:00
Aliaksandr Valialkin
fb9358635d lib/storage: mention source parts on merge error 
This should improve determining broken source part.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
 2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin
786beb8fc8 Makefile: enable golangci-lint in make check_all 2019-06-20 15:00:58 +03:00
Aliaksandr Valialkin
9cac11db64 lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:38:45 +03:00
Aliaksandr Valialkin
a0c22a6830 app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count 2019-06-19 18:37:38 +03:00
Aliaksandr Valialkin
f9e1d32168 lib/storage: persist metric ids for the current and the previous hour on graceful shutdown 
This should improve performance after restart when the db contains a lot of time series
with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)
 2019-06-14 07:55:09 +03:00
Aliaksandr Valialkin
18d6f293f7 lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func 
The func syncs parent dir in order to persist directory removal
in the event of power loss
 2019-06-12 01:55:18 +03:00
Aliaksandr Valialkin
51e2e255a6 lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error 2019-06-11 23:13:45 +03:00
Aliaksandr Valialkin
b491045a4b lib/{storage,mergeset}: sync filenames inside part when finalizing the part 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
 2019-06-11 21:51:19 +03:00
Aliaksandr Valialkin
3437c30180 all: try hard removing directory with contents 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
 2019-06-11 01:58:08 +03:00
Aliaksandr Valialkin
75a0acf72d app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values 2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin
0ccedbdfd2 lib/storage: mention the accountID and projectID in error message when filtering out other (accountID, projectID) entries 2019-06-10 14:43:53 +03:00
Aliaksandr Valialkin
d54f5fec0b lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter 
This should improve speed for searching metrics among high number of time series
with high churn rate like in big Kubernetes clusters with frequent deployments.
 2019-06-10 14:07:47 +03:00
Aliaksandr Valialkin
27e50e86f4 lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters 2019-06-10 13:26:00 +03:00
Aliaksandr Valialkin
b69d3dbd0c lib/storage: filter out metricIDs from another (AccountID, ProjectID) in getMetricIDsForRecentHours 2019-06-10 13:05:16 +03:00
Aliaksandr Valialkin
3059ae7be0 lib/storage: give clearer names to more functions 2019-06-10 12:59:33 +03:00
Aliaksandr Valialkin
d3a024d2d6 lib/storage: give more clear names to functions 2019-06-10 12:50:22 +03:00
Aliaksandr Valialkin
00e0760608 lib/storage: test GetSeriesCount 2019-06-10 12:40:33 +03:00
Aliaksandr Valialkin
e4cba5a7ed lib/storage: make getSeriesCount func indexSearch method 2019-06-10 12:29:24 +03:00
Aliaksandr Valialkin
d882afa905 lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes) 2019-06-09 19:14:04 +03:00
Aliaksandr Valialkin
a2986cde70 lib/storage: tune updating a map with today`s metric ids 
- Increase update iterval from 1s to 10s. This should reduce CPU usage
  for large amounts of metric ids with constant churn.
- Reduce pendingTodayMetricIDsLock lock duration during the update.
 2019-06-02 22:00:13 +03:00
Aliaksandr Valialkin
e27fd5148a lib/storage: speed up checking metricID existence in the list for the current date 2019-06-02 18:34:20 +03:00
Aliaksandr Valialkin
a6d02ff275 lib/timerpool: use timer pool in concurrency limiters 
This should reduce the number of memory allocations in highly loaded system
 2019-05-28 17:30:10 +03:00
Aliaksandr Valialkin
bdf696ef18 all: fix misspellings 2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
121a920a18 Makefile: add -s flag to go fmt in make fmt command 2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin
24578b4bb1 all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00