Commit Graph

752 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
8f3339fa81 app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin
6385432611 app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum 2020-11-10 01:18:59 +02:00
Roman Khavronenko
4fd2b6cd16 vmalert: explicitly set extra labels to alert entities (#886)
The previous implementation treated extra labels (global and rule labels) as
separate label set to returned time series labels. Hence, time series always contained
only original labels and alert ID was generated from sorted labels key-values.
Extra labels didn't affect the generated ID and were applied on the following actions:
- templating for Summary and Annotations;
- persisting state via remote write;
- restoring state via remote read.

Such behaviour caused difficulties on restore procedure because extra labels had to be dropped
before checking the alert ID, but that not always worked. Consider the case when expression
returns the following time series `up{job="foo"}` and rule has extra label `job=bar`.
This would mean that restored alert ID will be always different to the real time series because
of collision.

To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't
store original labels anymore. However, this could result into a new error situation.
Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`,
while rule has extra label `job=bar`. In such case, applying extra labels will result into
two identical time series and `vmalert` will return error:
 `result contains metrics with the same labelset after applying rule labels`

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
2020-11-10 00:27:56 +02:00
Aliaksandr Valialkin
a8562d643b lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin
b8083b7659 lib/promscrape: clean references to label name and label value strings after applying per-target relabeling
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin
efebc3b6fb app/vmselect/promql: code cleanup after 43823addea 2020-11-06 01:31:33 +02:00
n4mine
3127aa92b5 app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883) 2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin
767231f41f app/vmstorage/transport: properly handle request to labelValuesOnTimeRange 2020-11-05 02:08:04 +02:00
Aliaksandr Valialkin
72011bcc45 app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange 2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin
c5e6c5f5a6 app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set 2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin
1336e47c86 docs/vmagent.md: update after 4c808d58bf 2020-11-04 20:33:49 +02:00
Nikolay
5b235b902b Adds ready probe (#874)
* adds leading forward slash check for scrapeURL path
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835

* adds ready probe for scrape config initialization,
it should prevent metrics loss during vmagent rolling update,
/ready api will return 425 http code, if some scrape config still waits for initialization.

* updates docs

* Update app/vmagent/README.md

* renames var

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-04 20:33:48 +02:00
Nikolay
d0a9b24c5a reduces memory usage for vmagent, (#880)
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878

* Update app/vmagent/README.md

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin
407a46c11e Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels"
This reverts commit 1a80acc712.
2020-11-04 11:45:35 +02:00
Aliaksandr Valialkin
1a80acc712 docs/vmagent.md: mention about -promscrape.dropOriginalLabels 2020-11-04 11:16:16 +02:00
Aliaksandr Valialkin
887a3c317f app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop 2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin
66de02fbb4 app/vmselect/promql: allow dropping trailing sample only for default_rollup function
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin
ca2e0f1e04 app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code
This makes consistent the behaviour with Prometheus.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin
6b623eba02 app/vmselect/promql: go fmt 2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin
7c0b658865 app/vmselect/promql: do not drop trailing datapoints for instant queries
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-11-02 00:12:53 +02:00
Roman Khavronenko
333675875f vmalert: skip automatically added labels on alerts restore (#871)
Label `alertgroup` was introduced in #611 and automatically added to generated
time series. By mistake, this new label wasn't correctly purged on restore event
and affected alert's ID uniqueness. This commit removes `alertgroup` label
in restore function.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
2020-11-01 23:26:00 +02:00
kreedom
40172c0721 vmbackup fix panic when no origin fs given (#859)
* use fsnil when no origin fs
2020-11-01 23:17:01 +02:00
Aliaksandr Valialkin
ed724d25ba lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics 2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
abdf22e0bb app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets
This page is exposed by vmagent and by a single-node VictoriaMetrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin
c4464594b7 app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K 2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin
9c5cd5a6c5 lib/storage: code cleanup after 5bfd4e6218 2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin
0db7c2b500 app/vmstorage: support for -retentionPeriod smaller than one month
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-10-20 14:42:46 +03:00
kreedom
4526cf92d3 vmalert - add dryRun (#842)
vmalert: add `dryRun` flag for rules validation without running the service
2020-10-20 10:49:22 +03:00
Seva Poliakov
e6bf9eaac7 Fix typo in vnrestore readme 2020-10-20 10:49:22 +03:00
Aliaksandr Valialkin
ee2902ddaf app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series
Now trailing data points are additionally dropped for time series with a single raw sample

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-17 10:44:26 +03:00
Roman Khavronenko
d6155a3f33 vmalert: update docs to highlight the state restore requirements; (#833)
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830
2020-10-13 18:34:00 +03:00
Aliaksandr Valialkin
b9a4601c97 app/vmselect/promql: return a single time series at max from absent() function like Prometheus does 2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin
217c192c88 app/vmselect/promql: improve time series staleness detection
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin
f877e703c8 app/vmselect/promql: fix mode_over_time calculations
Previously `mode_over_time` could return garbage due to improper shuffling of input data points.
2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin
d884ab13dc app/vmselect/prometheus: fix golangci-lint warning 2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin
0867dea5fc app/vmselect: add ability to export data in CSV format via /api/v1/export/csv 2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin
938b3b7ed1 lib/promscrape: code prettifying after 9bd9f67718 2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin
7f96712b38 Adds dockerswarm sd (#818)
* adds dockerswarm service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656

 Following roles supported: services, tasks and nodes.
 Basic, token and tls auth supported.
 Added tests for labels generation.

* added unix socket support to discovery utils

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin
2d03d0e2dd app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning
Functions are:

* keep_last_value
* keep_next_value
* interpolate
* running_min
* running_max
* running_avg
* range_min
* range_max
* range_avg
* range_first
* range_last
* range_quantile
* smooth_exponential

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
3881c84afe Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus"
This reverts commit ac45082216.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric names:

* ceil
* floor
* round

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
79d70480b7 Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus"
This reverts commit bb61a4769b.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* clamp_min
* clamp_max

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
8c37b63ea9 Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus"
This reverts commit e5202a4eae.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* max_over_time
* min_over_time
* avg_over_time
* quantile_over_time
* geomean_over_time
* mode_over_time
* holt_winters
* predict_linear

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
de1c07b937 lib/backup: add MustStop() method for all remote filesystems 2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin
bf6d523bef lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter 2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin
9b7ce5d004 app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics 2020-10-09 13:38:49 +03:00
Aliaksandr Valialkin
d2e917d1cb app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start 2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin
4b1c401790 app/vmalert: accept days, weeks and years in for: part of config like Prometheus does
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817
2020-10-08 20:13:20 +03:00
Aliaksandr Valialkin
35b8ffaa17 docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors command-line flag usage 2020-10-08 19:24:05 +03:00
Aliaksandr Valialkin
0d44e371f3 lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.

Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin
f9f8e4a39c app/vmalert: do not pring description for all the flags on config errors
The description is too big to consume by human and it just distracts humans.
2020-10-08 13:35:46 +03:00