Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-23 12:31:07 +01:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity

app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs

Browse Source
This commit is contained in:
Aliaksandr Valialkin 2022-03-26 10:17:37 +02:00
parent a462b97859
commit 6e364e19ef
No known key found for this signature in database
GPG Key ID: A72BEC6CD3D0DED1
14 changed files with 132 additions and 66 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
README.md
View File

@ -820,13 +820,13 @@ Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series.
On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: On large databases you may experience problems with limit on the number of time series, which can be exported. In this case you need to adjust `-search.maxExportSeries` command-line flag:
```bash ```bash
# count unique timeseries in database # count unique timeseries in database
wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command # relaunch victoriametrics with search.maxExportSeries more than value from previous command
``` ```
Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either
@ -1835,6 +1835,12 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8) The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
-search.maxExportDuration duration -search.maxExportDuration duration
The maximum duration for /api/v1/export call (default 720h0m0s) The maximum duration for /api/v1/export call (default 720h0m0s)
-search.maxExportSeries int
The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage (default 1000000)
-search.maxFederateSeries int
The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 300000)
-search.maxGraphiteSeries int
The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
-search.maxLookback duration -search.maxLookback duration
Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
-search.maxPointsPerTimeseries int -search.maxPointsPerTimeseries int
@ -1850,12 +1856,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000) The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000)
-search.maxSamplesPerSeries int -search.maxSamplesPerSeries int
The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000) The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000)
-search.maxSeries int
The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage (default 10000)
-search.maxStalenessInterval duration -search.maxStalenessInterval duration
The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons
-search.maxStatusRequestDuration duration -search.maxStatusRequestDuration duration
The maximum duration for /api/v1/status/* requests (default 5m0s) The maximum duration for /api/v1/status/* requests (default 5m0s)
-search.maxStepForPointsAdjustment duration -search.maxStepForPointsAdjustment duration
The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s) The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s)
-search.maxTSDBStatusSeries int
The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage (default 1000000)
-search.maxTagKeys int -search.maxTagKeys int
The maximum number of tag keys returned from /api/v1/labels (default 100000) The maximum number of tag keys returned from /api/v1/labels (default 100000)
-search.maxTagValueSuffixesPerSearch int -search.maxTagValueSuffixesPerSearch int
@ -1863,7 +1873,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-search.maxTagValues int -search.maxTagValues int
The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000) The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000)
-search.maxUniqueTimeseries int -search.maxUniqueTimeseries int
The maximum number of unique time series each search can scan. This option allows limiting memory usage (default 300000) The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage (default 300000)
-search.minStalenessInterval duration -search.minStalenessInterval duration
The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval' The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval'
-search.noStaleMarkers -search.noStaleMarkers

12
app/vmselect/graphite/tags_api.go
View File

@ -54,7 +54,7 @@ func TagsDelSeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Re
}) })
} }
tfss := joinTagFilterss(tfs, etfs) tfss := joinTagFilterss(tfs, etfs)
sq := storage.NewSearchQuery(0, ct, tfss) sq := storage.NewSearchQuery(0, ct, tfss, 0)
n, err := netstorage.DeleteSeries(sq, deadline) n, err := netstorage.DeleteSeries(sq, deadline)
if err != nil { if err != nil {
return fmt.Errorf("cannot delete series for %q: %w", sq, err) return fmt.Errorf("cannot delete series for %q: %w", sq, err)
@ -196,7 +196,7 @@ func TagsAutoCompleteValuesHandler(startTime time.Time, w http.ResponseWriter, r
} }
} else { } else {
// Slow path: use netstorage.SearchMetricNames for applying `expr` filters. // Slow path: use netstorage.SearchMetricNames for applying `expr` filters.
sq, err := getSearchQueryForExprs(startTime, etfs, exprs) sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
if err != nil { if err != nil {
return err return err
} }
@ -282,7 +282,7 @@ func TagsAutoCompleteTagsHandler(startTime time.Time, w http.ResponseWriter, r *
} }
} else { } else {
// Slow path: use netstorage.SearchMetricNames for applying `expr` filters. // Slow path: use netstorage.SearchMetricNames for applying `expr` filters.
sq, err := getSearchQueryForExprs(startTime, etfs, exprs) sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
if err != nil { if err != nil {
return err return err
} }
@ -349,7 +349,7 @@ func TagsFindSeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.R
if err != nil { if err != nil {
return fmt.Errorf("cannot setup tag filters: %w", err) return fmt.Errorf("cannot setup tag filters: %w", err)
} }
sq, err := getSearchQueryForExprs(startTime, etfs, exprs) sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
if err != nil { if err != nil {
return err return err
} }
@ -474,14 +474,14 @@ func getInt(r *http.Request, argName string) (int, error) {
return n, nil return n, nil
} }
func getSearchQueryForExprs(startTime time.Time, etfs [][]storage.TagFilter, exprs []string) (*storage.SearchQuery, error) { func getSearchQueryForExprs(startTime time.Time, etfs [][]storage.TagFilter, exprs []string, maxMetrics int) (*storage.SearchQuery, error) {
tfs, err := exprsToTagFilters(exprs) tfs, err := exprsToTagFilters(exprs)
if err != nil { if err != nil {
return nil, err return nil, err
} }
ct := startTime.UnixNano() / 1e6 ct := startTime.UnixNano() / 1e6
tfss := joinTagFilterss(tfs, etfs) tfss := joinTagFilterss(tfs, etfs)
sq := storage.NewSearchQuery(0, ct, tfss) sq := storage.NewSearchQuery(0, ct, tfss, maxMetrics)
return sq, nil return sq, nil
} }

33
app/vmselect/netstorage/netstorage.go
View File

@ -26,7 +26,6 @@ var (
maxTagKeysPerSearch = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned from /api/v1/labels") maxTagKeysPerSearch = flag.Int("search.maxTagKeys", 100e3, "The maximum number of tag keys returned from /api/v1/labels")
maxTagValuesPerSearch = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned from /api/v1/label/<label_name>/values") maxTagValuesPerSearch = flag.Int("search.maxTagValues", 100e3, "The maximum number of tag values returned from /api/v1/label/<label_name>/values")
maxTagValueSuffixesPerSearch = flag.Int("search.maxTagValueSuffixesPerSearch", 100e3, "The maximum number of tag value suffixes returned from /metrics/find") maxTagValueSuffixesPerSearch = flag.Int("search.maxTagValueSuffixesPerSearch", 100e3, "The maximum number of tag value suffixes returned from /metrics/find")
maxMetricsPerSearch = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series each search can scan. This option allows limiting memory usage")
maxSamplesPerSeries = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage") maxSamplesPerSeries = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage")
maxSamplesPerQuery = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries") maxSamplesPerQuery = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries")
) )
@ -600,7 +599,7 @@ func DeleteSeries(sq *storage.SearchQuery, deadline searchutils.Deadline) (int,
MinTimestamp: sq.MinTimestamp, MinTimestamp: sq.MinTimestamp,
MaxTimestamp: sq.MaxTimestamp, MaxTimestamp: sq.MaxTimestamp,
} }
tfss, err := setupTfss(tr, sq.TagFilterss, deadline) tfss, err := setupTfss(tr, sq.TagFilterss, sq.MaxMetrics, deadline)
if err != nil { if err != nil {
return 0, err return 0, err
} }
@ -805,11 +804,11 @@ func GetLabelEntries(deadline searchutils.Deadline) ([]storage.TagEntry, error)
} }
// GetTSDBStatusForDate returns tsdb status according to https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats // GetTSDBStatusForDate returns tsdb status according to https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
func GetTSDBStatusForDate(deadline searchutils.Deadline, date uint64, topN int) (*storage.TSDBStatus, error) { func GetTSDBStatusForDate(deadline searchutils.Deadline, date uint64, topN, maxMetrics int) (*storage.TSDBStatus, error) {
if deadline.Exceeded() { if deadline.Exceeded() {
return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String()) return nil, fmt.Errorf("timeout exceeded before starting the query processing: %s", deadline.String())
} }
status, err := vmstorage.GetTSDBStatusForDate(date, topN, deadline.Deadline()) status, err := vmstorage.GetTSDBStatusForDate(date, topN, maxMetrics, deadline.Deadline())
if err != nil { if err != nil {
return nil, fmt.Errorf("error during tsdb status request: %w", err) return nil, fmt.Errorf("error during tsdb status request: %w", err)
} }
@ -827,12 +826,12 @@ func GetTSDBStatusWithFilters(deadline searchutils.Deadline, sq *storage.SearchQ
MinTimestamp: sq.MinTimestamp, MinTimestamp: sq.MinTimestamp,
MaxTimestamp: sq.MaxTimestamp, MaxTimestamp: sq.MaxTimestamp,
} }
tfss, err := setupTfss(tr, sq.TagFilterss, deadline) tfss, err := setupTfss(tr, sq.TagFilterss, sq.MaxMetrics, deadline)
if err != nil { if err != nil {
return nil, err return nil, err
} }
date := uint64(tr.MinTimestamp) / (3600 * 24 * 1000) date := uint64(tr.MinTimestamp) / (3600 * 24 * 1000)
status, err := vmstorage.GetTSDBStatusWithFiltersForDate(tfss, date, topN, deadline.Deadline()) status, err := vmstorage.GetTSDBStatusWithFiltersForDate(tfss, date, topN, sq.MaxMetrics, deadline.Deadline())
if err != nil { if err != nil {
return nil, fmt.Errorf("error during tsdb status with filters request: %w", err) return nil, fmt.Errorf("error during tsdb status with filters request: %w", err)
} }
@ -883,7 +882,7 @@ func ExportBlocks(sq *storage.SearchQuery, deadline searchutils.Deadline, f func
if err := vmstorage.CheckTimeRange(tr); err != nil { if err := vmstorage.CheckTimeRange(tr); err != nil {
return err return err
} }
tfss, err := setupTfss(tr, sq.TagFilterss, deadline) tfss, err := setupTfss(tr, sq.TagFilterss, sq.MaxMetrics, deadline)
if err != nil { if err != nil {
return err return err
} }
@ -894,7 +893,7 @@ func ExportBlocks(sq *storage.SearchQuery, deadline searchutils.Deadline, f func
sr := getStorageSearch() sr := getStorageSearch()
defer putStorageSearch(sr) defer putStorageSearch(sr)
startTime := time.Now() startTime := time.Now()
sr.Init(vmstorage.Storage, tfss, tr, *maxMetricsPerSearch, deadline.Deadline()) sr.Init(vmstorage.Storage, tfss, tr, sq.MaxMetrics, deadline.Deadline())
indexSearchDuration.UpdateDuration(startTime) indexSearchDuration.UpdateDuration(startTime)
// Start workers that call f in parallel on available CPU cores. // Start workers that call f in parallel on available CPU cores.
@ -991,12 +990,12 @@ func SearchMetricNames(sq *storage.SearchQuery, deadline searchutils.Deadline) (
if err := vmstorage.CheckTimeRange(tr); err != nil { if err := vmstorage.CheckTimeRange(tr); err != nil {
return nil, err return nil, err
} }
tfss, err := setupTfss(tr, sq.TagFilterss, deadline) tfss, err := setupTfss(tr, sq.TagFilterss, sq.MaxMetrics, deadline)
if err != nil { if err != nil {
return nil, err return nil, err
} }
mns, err := vmstorage.SearchMetricNames(tfss, tr, *maxMetricsPerSearch, deadline.Deadline()) mns, err := vmstorage.SearchMetricNames(tfss, tr, sq.MaxMetrics, deadline.Deadline())
if err != nil { if err != nil {
return nil, fmt.Errorf("cannot find metric names: %w", err) return nil, fmt.Errorf("cannot find metric names: %w", err)
} }
@ -1019,7 +1018,7 @@ func ProcessSearchQuery(sq *storage.SearchQuery, fetchData bool, deadline search
if err := vmstorage.CheckTimeRange(tr); err != nil { if err := vmstorage.CheckTimeRange(tr); err != nil {
return nil, err return nil, err
} }
tfss, err := setupTfss(tr, sq.TagFilterss, deadline) tfss, err := setupTfss(tr, sq.TagFilterss, sq.MaxMetrics, deadline)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -1029,7 +1028,7 @@ func ProcessSearchQuery(sq *storage.SearchQuery, fetchData bool, deadline search
sr := getStorageSearch() sr := getStorageSearch()
startTime := time.Now() startTime := time.Now()
maxSeriesCount := sr.Init(vmstorage.Storage, tfss, tr, *maxMetricsPerSearch, deadline.Deadline()) maxSeriesCount := sr.Init(vmstorage.Storage, tfss, tr, sq.MaxMetrics, deadline.Deadline())
indexSearchDuration.UpdateDuration(startTime) indexSearchDuration.UpdateDuration(startTime)
m := make(map[string][]blockRef, maxSeriesCount) m := make(map[string][]blockRef, maxSeriesCount)
orderedMetricNames := make([]string, 0, maxSeriesCount) orderedMetricNames := make([]string, 0, maxSeriesCount)
@ -1111,7 +1110,7 @@ type blockRef struct {
addr tmpBlockAddr addr tmpBlockAddr
} }
func setupTfss(tr storage.TimeRange, tagFilterss [][]storage.TagFilter, deadline searchutils.Deadline) ([]*storage.TagFilters, error) { func setupTfss(tr storage.TimeRange, tagFilterss [][]storage.TagFilter, maxMetrics int, deadline searchutils.Deadline) ([]*storage.TagFilters, error) {
tfss := make([]*storage.TagFilters, 0, len(tagFilterss)) tfss := make([]*storage.TagFilters, 0, len(tagFilterss))
for _, tagFilters := range tagFilterss { for _, tagFilters := range tagFilterss {
tfs := storage.NewTagFilters() tfs := storage.NewTagFilters()
@ -1119,13 +1118,13 @@ func setupTfss(tr storage.TimeRange, tagFilterss [][]storage.TagFilter, deadline
tf := &tagFilters[i] tf := &tagFilters[i]
if string(tf.Key) == "__graphite__" { if string(tf.Key) == "__graphite__" {
query := tf.Value query := tf.Value
paths, err := vmstorage.SearchGraphitePaths(tr, query, *maxMetricsPerSearch, deadline.Deadline()) paths, err := vmstorage.SearchGraphitePaths(tr, query, maxMetrics, deadline.Deadline())
if err != nil { if err != nil {
return nil, fmt.Errorf("error when searching for Graphite paths for query %q: %w", query, err) return nil, fmt.Errorf("error when searching for Graphite paths for query %q: %w", query, err)
} }
if len(paths) >= *maxMetricsPerSearch { if len(paths) >= maxMetrics {
return nil, fmt.Errorf("more than -search.maxUniqueTimeseries=%d time series match Graphite query %q; "+ return nil, fmt.Errorf("more than %d time series match Graphite query %q; "+
"either narrow down the query or increase -search.maxUniqueTimeseries command-line flag value", *maxMetricsPerSearch, query) "either narrow down the query or increase the corresponding -search.max* command-line flag value", maxMetrics, query)
} }
tfs.AddGraphiteQuery(query, paths, tf.IsNegative) tfs.AddGraphiteQuery(query, paths, tf.IsNegative)
continue continue

32
app/vmselect/prometheus/prometheus.go
View File

@ -42,6 +42,12 @@ var (
"See also '-search.maxLookback' flag, which has the same meaning due to historical reasons") "See also '-search.maxLookback' flag, which has the same meaning due to historical reasons")
maxStepForPointsAdjustment = flag.Duration("search.maxStepForPointsAdjustment", time.Minute, "The maximum step when /api/v1/query_range handler adjusts "+ maxStepForPointsAdjustment = flag.Duration("search.maxStepForPointsAdjustment", time.Minute, "The maximum step when /api/v1/query_range handler adjusts "+
"points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data") "points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data")
maxUniqueTimeseries = flag.Int("search.maxUniqueTimeseries", 300e3, "The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage")
maxFederateSeries = flag.Int("search.maxFederateSeries", 300e3, "The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage")
maxExportSeries = flag.Int("search.maxExportSeries", 1e6, "The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage")
maxTSDBStatusSeries = flag.Int("search.maxTSDBStatusSeries", 1e6, "The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage")
maxSeriesLimit = flag.Int("search.maxSeries", 10e3, "The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage")
) )
// Default step used if not set. // Default step used if not set.
@ -78,7 +84,7 @@ func FederateHandler(startTime time.Time, w http.ResponseWriter, r *http.Request
if err != nil { if err != nil {
return err return err
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxFederateSeries)
rss, err := netstorage.ProcessSearchQuery(sq, true, deadline) rss, err := netstorage.ProcessSearchQuery(sq, true, deadline)
if err != nil { if err != nil {
return fmt.Errorf("cannot fetch data for %q: %w", sq, err) return fmt.Errorf("cannot fetch data for %q: %w", sq, err)
@ -135,7 +141,7 @@ func ExportCSVHandler(startTime time.Time, w http.ResponseWriter, r *http.Reques
if err != nil { if err != nil {
return err return err
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxExportSeries)
w.Header().Set("Content-Type", "text/csv; charset=utf-8") w.Header().Set("Content-Type", "text/csv; charset=utf-8")
bw := bufferedwriter.Get(w) bw := bufferedwriter.Get(w)
defer bufferedwriter.Put(bw) defer bufferedwriter.Put(bw)
@ -232,7 +238,7 @@ func ExportNativeHandler(startTime time.Time, w http.ResponseWriter, r *http.Req
if err != nil { if err != nil {
return err return err
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxExportSeries)
w.Header().Set("Content-Type", "VictoriaMetrics/native") w.Header().Set("Content-Type", "VictoriaMetrics/native")
bw := bufferedwriter.Get(w) bw := bufferedwriter.Get(w)
defer bufferedwriter.Put(bw) defer bufferedwriter.Put(bw)
@ -383,7 +389,7 @@ func exportHandler(w http.ResponseWriter, matches []string, etfs [][]storage.Tag
} }
tagFilterss = searchutils.JoinTagFilterss(tagFilterss, etfs) tagFilterss = searchutils.JoinTagFilterss(tagFilterss, etfs)
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxExportSeries)
w.Header().Set("Content-Type", contentType) w.Header().Set("Content-Type", contentType)
bw := bufferedwriter.Get(w) bw := bufferedwriter.Get(w)
defer bufferedwriter.Put(bw) defer bufferedwriter.Put(bw)
@ -484,7 +490,7 @@ func DeleteHandler(startTime time.Time, r *http.Request) error {
return err return err
} }
ct := startTime.UnixNano() / 1e6 ct := startTime.UnixNano() / 1e6
sq := storage.NewSearchQuery(0, ct, tagFilterss) sq := storage.NewSearchQuery(0, ct, tagFilterss, 0)
deletedCount, err := netstorage.DeleteSeries(sq, deadline) deletedCount, err := netstorage.DeleteSeries(sq, deadline)
if err != nil { if err != nil {
return fmt.Errorf("cannot delete time series: %w", err) return fmt.Errorf("cannot delete time series: %w", err)
@ -597,7 +603,7 @@ func labelValuesWithMatches(labelName string, matches []string, etfs [][]storage
if len(tagFilterss) == 0 { if len(tagFilterss) == 0 {
logger.Panicf("BUG: tagFilterss must be non-empty") logger.Panicf("BUG: tagFilterss must be non-empty")
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxSeriesLimit)
m := make(map[string]struct{}) m := make(map[string]struct{})
if end-start > 24*3600*1000 { if end-start > 24*3600*1000 {
// It is cheaper to call SearchMetricNames on time ranges exceeding a day. // It is cheaper to call SearchMetricNames on time ranges exceeding a day.
@ -709,12 +715,12 @@ func TSDBStatusHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
} }
var status *storage.TSDBStatus var status *storage.TSDBStatus
if len(matches) == 0 && len(etfs) == 0 { if len(matches) == 0 && len(etfs) == 0 {
status, err = netstorage.GetTSDBStatusForDate(deadline, date, topN) status, err = netstorage.GetTSDBStatusForDate(deadline, date, topN, *maxTSDBStatusSeries)
if err != nil { if err != nil {
return fmt.Errorf(`cannot obtain tsdb status for date=%d, topN=%d: %w`, date, topN, err) return fmt.Errorf(`cannot obtain tsdb status for date=%d, topN=%d: %w`, date, topN, err)
} }
} else { } else {
status, err = tsdbStatusWithMatches(matches, etfs, date, topN, deadline) status, err = tsdbStatusWithMatches(matches, etfs, date, topN, *maxTSDBStatusSeries, deadline)
if err != nil { if err != nil {
return fmt.Errorf("cannot obtain tsdb status with matches for date=%d, topN=%d: %w", date, topN, err) return fmt.Errorf("cannot obtain tsdb status with matches for date=%d, topN=%d: %w", date, topN, err)
} }
@ -729,7 +735,7 @@ func TSDBStatusHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
return nil return nil
} }
func tsdbStatusWithMatches(matches []string, etfs [][]storage.TagFilter, date uint64, topN int, deadline searchutils.Deadline) (*storage.TSDBStatus, error) { func tsdbStatusWithMatches(matches []string, etfs [][]storage.TagFilter, date uint64, topN, maxMetrics int, deadline searchutils.Deadline) (*storage.TSDBStatus, error) {
tagFilterss, err := getTagFilterssFromMatches(matches) tagFilterss, err := getTagFilterssFromMatches(matches)
if err != nil { if err != nil {
return nil, err return nil, err
@ -740,7 +746,7 @@ func tsdbStatusWithMatches(matches []string, etfs [][]storage.TagFilter, date ui
} }
start := int64(date*secsPerDay) * 1000 start := int64(date*secsPerDay) * 1000
end := int64(date*secsPerDay+secsPerDay) * 1000 end := int64(date*secsPerDay+secsPerDay) * 1000
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, maxMetrics)
status, err := netstorage.GetTSDBStatusWithFilters(deadline, sq, topN) status, err := netstorage.GetTSDBStatusWithFilters(deadline, sq, topN)
if err != nil { if err != nil {
return nil, err return nil, err
@ -835,7 +841,7 @@ func labelsWithMatches(matches []string, etfs [][]storage.TagFilter, start, end
if len(tagFilterss) == 0 { if len(tagFilterss) == 0 {
logger.Panicf("BUG: tagFilterss must be non-empty") logger.Panicf("BUG: tagFilterss must be non-empty")
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxSeriesLimit)
m := make(map[string]struct{}) m := make(map[string]struct{})
if end-start > 24*3600*1000 { if end-start > 24*3600*1000 {
// It is cheaper to call SearchMetricNames on time ranges exceeding a day. // It is cheaper to call SearchMetricNames on time ranges exceeding a day.
@ -933,7 +939,7 @@ func SeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
if start >= end { if start >= end {
end = start + defaultStep end = start + defaultStep
} }
sq := storage.NewSearchQuery(start, end, tagFilterss) sq := storage.NewSearchQuery(start, end, tagFilterss, *maxSeriesLimit)
if end-start > 24*3600*1000 { if end-start > 24*3600*1000 {
// It is cheaper to call SearchMetricNames on time ranges exceeding a day. // It is cheaper to call SearchMetricNames on time ranges exceeding a day.
mns, err := netstorage.SearchMetricNames(sq, deadline) mns, err := netstorage.SearchMetricNames(sq, deadline)
@ -1080,6 +1086,7 @@ func QueryHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) e
Start: start, Start: start,
End: start, End: start,
Step: step, Step: step,
MaxSeries: *maxUniqueTimeseries,
QuotedRemoteAddr: httpserver.GetQuotedRemoteAddr(r), QuotedRemoteAddr: httpserver.GetQuotedRemoteAddr(r),
Deadline: deadline, Deadline: deadline,
LookbackDelta: lookbackDelta, LookbackDelta: lookbackDelta,
@ -1170,6 +1177,7 @@ func queryRangeHandler(startTime time.Time, w http.ResponseWriter, query string,
Start: start, Start: start,
End: end, End: end,
Step: step, Step: step,
MaxSeries: *maxUniqueTimeseries,
QuotedRemoteAddr: httpserver.GetQuotedRemoteAddr(r), QuotedRemoteAddr: httpserver.GetQuotedRemoteAddr(r),
Deadline: deadline, Deadline: deadline,
MayCache: mayCache, MayCache: mayCache,

19
app/vmselect/promql/eval.go
View File

@ -93,6 +93,10 @@ type EvalConfig struct {
End int64 End int64
Step int64 Step int64
// MaxSeries is the maximum number of time series, which can be scanned by the query.
// Zero means 'no limit'
MaxSeries int
// QuotedRemoteAddr contains quoted remote address. // QuotedRemoteAddr contains quoted remote address.
QuotedRemoteAddr string QuotedRemoteAddr string
@ -113,12 +117,13 @@ type EvalConfig struct {
timestampsOnce sync.Once timestampsOnce sync.Once
} }
// newEvalConfig returns new EvalConfig copy from src. // copyEvalConfig returns src copy.
func newEvalConfig(src *EvalConfig) *EvalConfig { func copyEvalConfig(src *EvalConfig) *EvalConfig {
var ec EvalConfig var ec EvalConfig
ec.Start = src.Start ec.Start = src.Start
ec.End = src.End ec.End = src.End
ec.Step = src.Step ec.Step = src.Step
ec.MaxSeries = src.MaxSeries
ec.Deadline = src.Deadline ec.Deadline = src.Deadline
ec.MayCache = src.MayCache ec.MayCache = src.MayCache
ec.LookbackDelta = src.LookbackDelta ec.LookbackDelta = src.LookbackDelta
@ -575,7 +580,7 @@ func evalRollupFunc(ec *EvalConfig, funcName string, rf rollupFunc, expr metrics
return nil, fmt.Errorf("`@` modifier must return a single series; it returns %d series instead", len(tssAt)) return nil, fmt.Errorf("`@` modifier must return a single series; it returns %d series instead", len(tssAt))
} }
atTimestamp := int64(tssAt[0].Values[0] * 1000) atTimestamp := int64(tssAt[0].Values[0] * 1000)
ecNew := newEvalConfig(ec) ecNew := copyEvalConfig(ec)
ecNew.Start = atTimestamp ecNew.Start = atTimestamp
ecNew.End = atTimestamp ecNew.End = atTimestamp
tss, err := evalRollupFuncWithoutAt(ecNew, funcName, rf, expr, re, iafc) tss, err := evalRollupFuncWithoutAt(ecNew, funcName, rf, expr, re, iafc)
@ -602,7 +607,7 @@ func evalRollupFuncWithoutAt(ec *EvalConfig, funcName string, rf rollupFunc, exp
var offset int64 var offset int64
if re.Offset != nil { if re.Offset != nil {
offset = re.Offset.Duration(ec.Step) offset = re.Offset.Duration(ec.Step)
ecNew = newEvalConfig(ecNew) ecNew = copyEvalConfig(ecNew)
ecNew.Start -= offset ecNew.Start -= offset
ecNew.End -= offset ecNew.End -= offset
// There is no need in calling AdjustStartEnd() on ecNew if ecNew.MayCache is set to true, // There is no need in calling AdjustStartEnd() on ecNew if ecNew.MayCache is set to true,
@ -615,7 +620,7 @@ func evalRollupFuncWithoutAt(ec *EvalConfig, funcName string, rf rollupFunc, exp
// in order to obtain expected OHLC results. // in order to obtain expected OHLC results.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309#issuecomment-582113462 // See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309#issuecomment-582113462
step := ecNew.Step step := ecNew.Step
ecNew = newEvalConfig(ecNew) ecNew = copyEvalConfig(ecNew)
ecNew.Start += step ecNew.Start += step
ecNew.End += step ecNew.End += step
offset -= step offset -= step
@ -679,7 +684,7 @@ func evalRollupFuncWithSubquery(ec *EvalConfig, funcName string, rf rollupFunc,
} }
window := re.Window.Duration(ec.Step) window := re.Window.Duration(ec.Step)
ecSQ := newEvalConfig(ec) ecSQ := copyEvalConfig(ec)
ecSQ.Start -= window + maxSilenceInterval + step ecSQ.Start -= window + maxSilenceInterval + step
ecSQ.End += step ecSQ.End += step
ecSQ.Step = step ecSQ.Step = step
@ -834,7 +839,7 @@ func evalRollupFuncWithMetricExpr(ec *EvalConfig, funcName string, rf rollupFunc
} else { } else {
minTimestamp -= ec.Step minTimestamp -= ec.Step
} }
sq := storage.NewSearchQuery(minTimestamp, ec.End, tfss) sq := storage.NewSearchQuery(minTimestamp, ec.End, tfss, ec.MaxSeries)
rss, err := netstorage.ProcessSearchQuery(sq, true, ec.Deadline) rss, err := netstorage.ProcessSearchQuery(sq, true, ec.Deadline)
if err != nil { if err != nil {
return nil, err return nil, err

2
app/vmselect/promql/exec_test.go
View File

@ -61,6 +61,7 @@ func TestExecSuccess(t *testing.T) {
Start: start, Start: start,
End: end, End: end,
Step: step, Step: step,
MaxSeries: 1000,
Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""), Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""),
RoundDigits: 100, RoundDigits: 100,
} }
@ -7496,6 +7497,7 @@ func TestExecError(t *testing.T) {
Start: 1000, Start: 1000,
End: 2000, End: 2000,
Step: 100, Step: 100,
MaxSeries: 1000,
Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""), Deadline: searchutils.NewDeadline(time.Now(), time.Minute, ""),
RoundDigits: 100, RoundDigits: 100,
} }

8
app/vmstorage/main.go
View File

@ -227,17 +227,17 @@ func SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]storage.
} }
// GetTSDBStatusForDate returns TSDB status for the given date. // GetTSDBStatusForDate returns TSDB status for the given date.
func GetTSDBStatusForDate(date uint64, topN int, deadline uint64) (*storage.TSDBStatus, error) { func GetTSDBStatusForDate(date uint64, topN, maxMetrics int, deadline uint64) (*storage.TSDBStatus, error) {
WG.Add(1) WG.Add(1)
status, err := Storage.GetTSDBStatusWithFiltersForDate(nil, date, topN, deadline) status, err := Storage.GetTSDBStatusWithFiltersForDate(nil, date, topN, maxMetrics, deadline)
WG.Done() WG.Done()
return status, err return status, err
} }
// GetTSDBStatusWithFiltersForDate returns TSDB status for given filters on the given date. // GetTSDBStatusWithFiltersForDate returns TSDB status for given filters on the given date.
func GetTSDBStatusWithFiltersForDate(tfss []*storage.TagFilters, date uint64, topN int, deadline uint64) (*storage.TSDBStatus, error) { func GetTSDBStatusWithFiltersForDate(tfss []*storage.TagFilters, date uint64, topN, maxMetrics int, deadline uint64) (*storage.TSDBStatus, error) {
WG.Add(1) WG.Add(1)
status, err := Storage.GetTSDBStatusWithFiltersForDate(tfss, date, topN, deadline) status, err := Storage.GetTSDBStatusWithFiltersForDate(tfss, date, topN, maxMetrics, deadline)
WG.Done() WG.Done()
return status, err return status, err
} }

12
docs/CHANGELOG.md
View File

@ -15,6 +15,18 @@ The following tip changes can be tested by building VictoriaMetrics components f
## tip ## tip
* FEATURE: add the following command-line flags, which can be used for fine-grained limiting of CPU and memory usage during various API calls:
* `-search.maxFederateSeries` for limiting the number of time series, which can be returned from [/federate](https://docs.victoriametrics.com/#federation).
* `-search.maxExportSeries` for limiting the number of time series, which can be returned from [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-time-series).
* `-search.maxSeries` for limiting the number of time series, which can be returned from [/api/v1/series](https://docs.victoriametrics.com/url-examples.html#apiv1series).
* `-search.maxTSDBStatusSeries` for limiting the number of time series, which can be scanned during the request to [/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats).
* `-search.maxGraphiteSeries` for limiting the number of time series, which can be scanned during the request to [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage).
Previously the `-search.maxUniqueTimeseries` command-line flag was used as a global limit for all these APIs. Now the `-search.maxUniqueTimeseries` is used only for limiting the number of time series, which can be scanned during requests to [/api/v1/query](https://docs.victoriametrics.com/url-examples.html#apiv1query) and [/api/v1/query_range](https://docs.victoriametrics.com/url-examples.html#apiv1query_range).
When using [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html), these command-line flags (including `-search.maxUniqueTimeseries`) must be passed to `vmselect` instead of `vmstorage`.
* BUGFIX: return `Content-Type: text/html` response header when requesting `/` HTTP path at VictoriaMetrics components. Previously `text/plain` response header was returned, which could lead to broken page formatting. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2323). * BUGFIX: return `Content-Type: text/html` response header when requesting `/` HTTP path at VictoriaMetrics components. Previously `text/plain` response header was returned, which could lead to broken page formatting. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2323).
## [v1.75.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.75.0) ## [v1.75.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.75.0)

16
docs/README.md
View File

@ -820,13 +820,13 @@ Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series.
On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: On large databases you may experience problems with limit on the number of time series, which can be exported. In this case you need to adjust `-search.maxExportSeries` command-line flag:
```bash ```bash
# count unique timeseries in database # count unique timeseries in database
wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command # relaunch victoriametrics with search.maxExportSeries more than value from previous command
``` ```
Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either
@ -1835,6 +1835,12 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8) The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
-search.maxExportDuration duration -search.maxExportDuration duration
The maximum duration for /api/v1/export call (default 720h0m0s) The maximum duration for /api/v1/export call (default 720h0m0s)
-search.maxExportSeries int
The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage (default 1000000)
-search.maxFederateSeries int
The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 300000)
-search.maxGraphiteSeries int
The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
-search.maxLookback duration -search.maxLookback duration
Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
-search.maxPointsPerTimeseries int -search.maxPointsPerTimeseries int
@ -1850,12 +1856,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000) The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000)
-search.maxSamplesPerSeries int -search.maxSamplesPerSeries int
The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000) The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000)
-search.maxSeries int
The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage (default 10000)
-search.maxStalenessInterval duration -search.maxStalenessInterval duration
The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons
-search.maxStatusRequestDuration duration -search.maxStatusRequestDuration duration
The maximum duration for /api/v1/status/* requests (default 5m0s) The maximum duration for /api/v1/status/* requests (default 5m0s)
-search.maxStepForPointsAdjustment duration -search.maxStepForPointsAdjustment duration
The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s) The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s)
-search.maxTSDBStatusSeries int
The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage (default 1000000)
-search.maxTagKeys int -search.maxTagKeys int
The maximum number of tag keys returned from /api/v1/labels (default 100000) The maximum number of tag keys returned from /api/v1/labels (default 100000)
-search.maxTagValueSuffixesPerSearch int -search.maxTagValueSuffixesPerSearch int
@ -1863,7 +1873,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-search.maxTagValues int -search.maxTagValues int
The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000) The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000)
-search.maxUniqueTimeseries int -search.maxUniqueTimeseries int
The maximum number of unique time series each search can scan. This option allows limiting memory usage (default 300000) The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage (default 300000)
-search.minStalenessInterval duration -search.minStalenessInterval duration
The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval' The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval'
-search.noStaleMarkers -search.noStaleMarkers

16
docs/Single-server-VictoriaMetrics.md
View File

@ -824,13 +824,13 @@ Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series. for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series.
On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter: On large databases you may experience problems with limit on the number of time series, which can be exported. In this case you need to adjust `-search.maxExportSeries` command-line flag:
```bash ```bash
# count unique timeseries in database # count unique timeseries in database
wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]' wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command # relaunch victoriametrics with search.maxExportSeries more than value from previous command
``` ```
Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either
@ -1839,6 +1839,12 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8) The maximum number of concurrent search requests. It shouldn't be high, since a single request can saturate all the CPU cores. See also -search.maxQueueDuration (default 8)
-search.maxExportDuration duration -search.maxExportDuration duration
The maximum duration for /api/v1/export call (default 720h0m0s) The maximum duration for /api/v1/export call (default 720h0m0s)
-search.maxExportSeries int
The maximum number of time series, which can be returned from /api/v1/export* APIs. This option allows limiting memory usage (default 1000000)
-search.maxFederateSeries int
The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 300000)
-search.maxGraphiteSeries int
The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
-search.maxLookback duration -search.maxLookback duration
Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
-search.maxPointsPerTimeseries int -search.maxPointsPerTimeseries int
@ -1854,12 +1860,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000) The maximum number of raw samples a single query can process across all time series. This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries (default 1000000000)
-search.maxSamplesPerSeries int -search.maxSamplesPerSeries int
The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000) The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage (default 30000000)
-search.maxSeries int
The maximum number of time series, which can be returned from /api/v1/series. This option allows limiting memory usage (default 10000)
-search.maxStalenessInterval duration -search.maxStalenessInterval duration
The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons The maximum interval for staleness calculations. By default it is automatically calculated from the median interval between samples. This flag could be useful for tuning Prometheus data model closer to Influx-style data model. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.maxLookback' flag, which has the same meaning due to historical reasons
-search.maxStatusRequestDuration duration -search.maxStatusRequestDuration duration
The maximum duration for /api/v1/status/* requests (default 5m0s) The maximum duration for /api/v1/status/* requests (default 5m0s)
-search.maxStepForPointsAdjustment duration -search.maxStepForPointsAdjustment duration
The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s) The maximum step when /api/v1/query_range handler adjusts points with timestamps closer than -search.latencyOffset to the current time. The adjustment is needed because such points may contain incomplete data (default 1m0s)
-search.maxTSDBStatusSeries int
The maximum number of time series, which can be processed during the call to /api/v1/status/tsdb. This option allows limiting memory usage (default 1000000)
-search.maxTagKeys int -search.maxTagKeys int
The maximum number of tag keys returned from /api/v1/labels (default 100000) The maximum number of tag keys returned from /api/v1/labels (default 100000)
-search.maxTagValueSuffixesPerSearch int -search.maxTagValueSuffixesPerSearch int
@ -1867,7 +1877,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
-search.maxTagValues int -search.maxTagValues int
The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000) The maximum number of tag values returned from /api/v1/label/<label_name>/values (default 100000)
-search.maxUniqueTimeseries int -search.maxUniqueTimeseries int
The maximum number of unique time series each search can scan. This option allows limiting memory usage (default 300000) The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage (default 300000)
-search.minStalenessInterval duration -search.minStalenessInterval duration
The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval' The minimum interval for staleness calculations. This flag could be useful for removing gaps on graphs generated from time series with irregular intervals between samples. See also '-search.maxStalenessInterval'
-search.noStaleMarkers -search.noStaleMarkers

10
lib/storage/index_db.go
View File

@ -1315,9 +1315,9 @@ func (is *indexSearch) getSeriesCount() (uint64, error) {
} }
// GetTSDBStatusWithFiltersForDate returns topN entries for tsdb status for the given tfss and the given date. // GetTSDBStatusWithFiltersForDate returns topN entries for tsdb status for the given tfss and the given date.
func (db *indexDB) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN int, deadline uint64) (*TSDBStatus, error) { func (db *indexDB) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN, maxMetrics int, deadline uint64) (*TSDBStatus, error) {
is := db.getIndexSearch(deadline) is := db.getIndexSearch(deadline)
status, err := is.getTSDBStatusWithFiltersForDate(tfss, date, topN) status, err := is.getTSDBStatusWithFiltersForDate(tfss, date, topN, maxMetrics)
db.putIndexSearch(is) db.putIndexSearch(is)
if err != nil { if err != nil {
return nil, err return nil, err
@ -1327,7 +1327,7 @@ func (db *indexDB) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint
} }
ok := db.doExtDB(func(extDB *indexDB) { ok := db.doExtDB(func(extDB *indexDB) {
is := extDB.getIndexSearch(deadline) is := extDB.getIndexSearch(deadline)
status, err = is.getTSDBStatusWithFiltersForDate(tfss, date, topN) status, err = is.getTSDBStatusWithFiltersForDate(tfss, date, topN, maxMetrics)
extDB.putIndexSearch(is) extDB.putIndexSearch(is)
}) })
if ok && err != nil { if ok && err != nil {
@ -1337,14 +1337,14 @@ func (db *indexDB) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint
} }
// getTSDBStatusWithFiltersForDate returns topN entries for tsdb status for the given tfss and the given date. // getTSDBStatusWithFiltersForDate returns topN entries for tsdb status for the given tfss and the given date.
func (is *indexSearch) getTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN int) (*TSDBStatus, error) { func (is *indexSearch) getTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN, maxMetrics int) (*TSDBStatus, error) {
var filter *uint64set.Set var filter *uint64set.Set
if len(tfss) > 0 { if len(tfss) > 0 {
tr := TimeRange{ tr := TimeRange{
MinTimestamp: int64(date) * msecPerDay, MinTimestamp: int64(date) * msecPerDay,
MaxTimestamp: int64(date+1) * msecPerDay, MaxTimestamp: int64(date+1) * msecPerDay,
} }
metricIDs, err := is.searchMetricIDsInternal(tfss, tr, 2e9) metricIDs, err := is.searchMetricIDsInternal(tfss, tr, maxMetrics)
if err != nil { if err != nil {
return nil, err return nil, err
} }

4
lib/storage/index_db_test.go
View File

@ -1766,7 +1766,7 @@ func TestSearchTSIDWithTimeRange(t *testing.T) {
} }
// Check GetTSDBStatusWithFiltersForDate with nil filters. // Check GetTSDBStatusWithFiltersForDate with nil filters.
status, err := db.GetTSDBStatusWithFiltersForDate(nil, baseDate, 5, noDeadline) status, err := db.GetTSDBStatusWithFiltersForDate(nil, baseDate, 5, 1e6, noDeadline)
if err != nil { if err != nil {
t.Fatalf("error in GetTSDBStatusWithFiltersForDate with nil filters: %s", err) t.Fatalf("error in GetTSDBStatusWithFiltersForDate with nil filters: %s", err)
} }
@ -1834,7 +1834,7 @@ func TestSearchTSIDWithTimeRange(t *testing.T) {
if err := tfs.Add([]byte("day"), []byte("0"), false, false); err != nil { if err := tfs.Add([]byte("day"), []byte("0"), false, false); err != nil {
t.Fatalf("cannot add filter: %s", err) t.Fatalf("cannot add filter: %s", err)
} }
status, err = db.GetTSDBStatusWithFiltersForDate([]*TagFilters{tfs}, baseDate, 5, noDeadline) status, err = db.GetTSDBStatusWithFiltersForDate([]*TagFilters{tfs}, baseDate, 5, 1e6, noDeadline)
if err != nil { if err != nil {
t.Fatalf("error in GetTSDBStatusWithFiltersForDate: %s", err) t.Fatalf("error in GetTSDBStatusWithFiltersForDate: %s", err)
} }

14
lib/storage/search.go
View File

@ -226,17 +226,27 @@ func (s *Search) NextMetricBlock() bool {
// SearchQuery is used for sending search queries from vmselect to vmstorage. // SearchQuery is used for sending search queries from vmselect to vmstorage.
type SearchQuery struct { type SearchQuery struct {
// The time range for searching time series
MinTimestamp int64 MinTimestamp int64
MaxTimestamp int64 MaxTimestamp int64
TagFilterss [][]TagFilter
// Tag filters for the search query
TagFilterss [][]TagFilter
// The maximum number of time series the search query can return.
MaxMetrics int
} }
// NewSearchQuery creates new search query for the given args. // NewSearchQuery creates new search query for the given args.
func NewSearchQuery(start, end int64, tagFilterss [][]TagFilter) *SearchQuery { func NewSearchQuery(start, end int64, tagFilterss [][]TagFilter, maxMetrics int) *SearchQuery {
if maxMetrics <= 0 {
maxMetrics = 2e9
}
return &SearchQuery{ return &SearchQuery{
MinTimestamp: start, MinTimestamp: start,
MaxTimestamp: end, MaxTimestamp: end,
TagFilterss: tagFilterss, TagFilterss: tagFilterss,
MaxMetrics: maxMetrics,
} }
} }

4
lib/storage/storage.go
View File

@ -1468,8 +1468,8 @@ func (s *Storage) GetSeriesCount(deadline uint64) (uint64, error) {
} }
// GetTSDBStatusWithFiltersForDate returns TSDB status data for /api/v1/status/tsdb with match[] filters. // GetTSDBStatusWithFiltersForDate returns TSDB status data for /api/v1/status/tsdb with match[] filters.
func (s *Storage) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN int, deadline uint64) (*TSDBStatus, error) { func (s *Storage) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN, maxMetrics int, deadline uint64) (*TSDBStatus, error) {
return s.idb().GetTSDBStatusWithFiltersForDate(tfss, date, topN, deadline) return s.idb().GetTSDBStatusWithFiltersForDate(tfss, date, topN, maxMetrics, deadline)
} }
// MetricRow is a metric to insert into storage. // MetricRow is a metric to insert into storage.