Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-30 07:40:06 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,916 Commits 113 Branches 748 Tags 745 MiB
59e7755df9
Commit Graph

125 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
84fb59b0ba lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:04:30 +03:00
Aliaksandr Valialkin
c4f3fbfa5d lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:42:28 +03:00
Aliaksandr Valialkin
2d8bd41f8a lib/storage: reduce memory allocations when syncing dateMetricIDCache 2021-06-03 16:20:42 +03:00
Aliaksandr Valialkin
39ef1e7a51 lib/storage: do not stop data ingestion on the first error in Storage.AddRows 
Continue data ingestion for the rest of blocks.
 2021-05-24 15:32:47 +03:00
Aliaksandr Valialkin
4b01c9fb2e lib/storage: limit the number of rows per each block in Storage.AddRows() 
This should reduce memory usage when ingesting big blocks or rows.
 2021-05-24 15:24:07 +03:00
Aliaksandr Valialkin
f54133b200 lib/storage: do not populate MetricID->MetricName cache during data ingestion 
This cache isn't needed during data ingestion, so there is no need in spending RAM on it.

This reduces RAM usage on data ingestion path by 30%
 2021-05-24 03:02:46 +03:00
Aliaksandr Valialkin
ad73f226ff app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 14:15:19 +03:00
Aliaksandr Valialkin
d7be2753c0 lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss 2021-05-13 09:02:33 +03:00
Aliaksandr Valialkin
832651c6c2 app/vmselect: follow up after 8a0678678b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168
 2021-05-12 17:18:30 +03:00
Nikolay
8a0678678b
Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin
12d733dd5d app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:52:57 +03:00
Aliaksandr Valialkin
dc9eafcd02 app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage 
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
 2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin
e1f699bb6c lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels 2021-03-31 21:24:46 +03:00
Aliaksandr Valialkin
aa81039b42 app/vmselect: log the metric which trigger rollup result cache reset 
This should help finding the source of stale metrics
 2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
3cfb3a3683 lib/storage: respect the deadline passed to Storage.SearchMetricNames 2021-03-22 23:03:17 +02:00
Aliaksandr Valialkin
8e2afdf568 lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache 2021-03-22 22:49:18 +02:00
Aliaksandr Valialkin
726f6ad804 lib/storage: small code simplification after 6cee5338b2 2021-03-18 15:21:13 +02:00
Aliaksandr Valialkin
6cee5338b2 lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars 
The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137
 2021-03-18 14:53:47 +02:00
John Belmonte
364fdf4a56
spelling fix: adjacent (#1115) 2021-03-09 09:18:19 +02:00
Aliaksandr Valialkin
4a07820048 lib/storage: make sure that nobody uses partitions when closing the table 2021-02-17 14:59:04 +02:00
Aliaksandr Valialkin
4e39bf148c vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
9f5ac603a7 lib/storage: reduce the minimum supported retention for inverted index from one month to one day 2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
57cac289e0 lib/storage: fix inconsistencies in error logs 2021-02-10 18:12:16 +02:00
Aliaksandr Valialkin
5d5f0b0627 lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata 2021-02-10 17:55:40 +02:00
Aliaksandr Valialkin
553016ea99 lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
6b4e6c229c lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day 
This should help systems with multiple CPU cores
 2021-02-10 00:01:13 +02:00
Aliaksandr Valialkin
d56390b925 optimize Storage.updatePerDateData() 2021-02-09 02:55:36 +02:00
Aliaksandr Valialkin
2242647a04 lib/storage: optimize data ingestion in the beginning of every hour 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-08 12:01:12 +02:00
Aliaksandr Valialkin
83d3e582ab lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion 
This should reduce possible CPU usage spikes at the beginning of every hour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-04 18:48:13 +02:00
Aliaksandr Valialkin
d16f22f3a1 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
a5a1b9bd66 lib/storage: fix a bug, which breaks searching by Graphite wildcard filters 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
157c02622b app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
8a057e705a lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention 
This should simplify debugging when the source of the metric with unexpected timestamp must be found.
 2020-11-25 14:41:37 +02:00
Aliaksandr Valialkin
b65236530c lib/storage: typo fix in error message: allowd->allowed 2020-11-25 14:15:42 +02:00
Aliaksandr Valialkin
465923b181 app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
 2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
48d033a198 app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
 2020-11-16 02:39:58 +02:00
immerrr again
51c529a2b6
app/vmstorage: add "/internal/force_flush" endpoint (#893) 2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin
b378cd6ed8 app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set 2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
fe289331dd lib/storage: remove obsolete code 2020-11-02 19:11:59 +02:00
Aliaksandr Valialkin
64e2d66014 lib/storage: code cleanup after 5bfd4e6218 2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
5bfd4e6218 app/vmstorage: support for -retentionPeriod smaller than one month 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
 2020-10-20 14:31:44 +03:00
Aliaksandr Valialkin
68f0e00761 app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start 2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin
764dc2499f lib/storage: code cleanup after 10f2eedee0 
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.

While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
 2020-10-01 19:06:23 +03:00
Aliaksandr Valialkin
26115891db lib/decimal: properly store Inf values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-18 19:07:07 +03:00
Aliaksandr Valialkin
1f33dd717f lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions 
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin
5a90a92378 lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-11 14:44:53 +03:00
Aliaksandr Valialkin
f6bc608e86 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
9d8fdff6c5 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin
582c74cd93 lib/storage: mention tag filters used in the query that led to error message 
This should improve detecting invalid or heavy queries that lead to errors.
 2020-08-10 13:36:49 +03:00