Commit Graph

1924 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
4d27fa41c8
Makefile: run errcheck for all the app/... subdirs 2022-09-30 18:35:53 +03:00
Nikolay
33f40f4a5f
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 18:35:53 +03:00
Roman Khavronenko
740bb2cc00
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 17:10:17 +02:00
Dmytro Kozlov
e220bc3cd5
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:55:12 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:20:34 +03:00
Roman Khavronenko
39f559d22b
vmalert: allow using extra labels in annotations (#3181)
According to Ruler specification, only labels returned within time series
should be available for use in annotations.

For long time, vmalert didn't respect this rule. And in PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2403
this was fixed for the sake of compatibility. However, this resulted
into users confusion, as they expected all configured and extra labels
to be available - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013

This fix allows to use extra labels in Annotations. But in the case of conflicts
the original labels (extracted from time series) are preferred.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-29 18:22:50 +02:00
Aliaksandr Valialkin
ef435f8cc4
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
Optimize this function by using results cache for input strings.
Use this function all over the code.

This is a follow-up for fcffdba9dc

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:40:59 +03:00
panguicai
fbc85e654c
docs: fix typo for vmalert docs (#3173)
Signed-off-by: panguicai008 <1121906548@qq.com>
2022-09-28 09:15:05 +02:00
Dmytro Kozlov
6ab1cede62
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:35:45 +03:00
Roman Khavronenko
166d444159
vmselect/rollup: rm workaround for slow-changing counters (#3163)
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 15:33:25 +03:00
Aliaksandr Valialkin
7c2474dac7
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:57:23 +03:00
Aliaksandr Valialkin
fcffdba9dc
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:11:39 +03:00
Aliaksandr Valialkin
819aa95552
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-09-26 08:47:31 +03:00
Aliaksandr Valialkin
6b71f33c8b
docs/vmalert.md: follow-up after 7748a9d629
- Consistently use single dash in front of command-line flags instead of double dashes.
- Add a warning that too small -search.latencyOffset may lead to incomplete query results.
2022-09-26 08:36:24 +03:00
Roman Khavronenko
0c95f928ae
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-23 10:22:53 +02:00
Roman Khavronenko
7748a9d629
vmalert: add info about search.latencyOffset to Troubleshooting (#3151)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-23 09:49:14 +02:00
Dmytro Kozlov
6a794ec5d5
app/{vmctl,vmalert}: update progress bar library (make vendor-update) (#3138)
* app/{vmctl,vmalert}: update progress bar library (make vendor-update)

* app/{vmctl,vmalert}: make vendor-update
2022-09-21 11:08:33 +03:00
Roman Khavronenko
d61cce06fd
vmalert: prodvide more details on duplicates (#3136)
Now vmalert will print the following messages on dupliсates:
```
"recording rule \"record\"; expr: \"up == 1\"; labels: summary={{ value|query }}" is a duplicate within the group "test"
"alerting rule \"alert\"; expr: \"up == 1\"; labels: description={{ value|query }}" is a duplicate within the group "test"
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3127
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-20 12:52:46 +02:00
Aliaksandr Valialkin
5e2fcd455d
app/vmagent/remotewrite: go fmt after 2b55d167d7 2022-09-19 15:14:35 +03:00
Aliaksandr Valialkin
2b55d167d7
app/vmagent/remotewrite: add benchmarks for comparing the performance of standard Snappy encoder with github.com/klauspost/compress/s2 encoder
The standard Snappy encoder from github.com/golang/snappy shows quite good performance number
for compressing the Prometheus remote_write proto messages according to the added benchmarks,
so there is no need in switching to github.com/klauspost/compress/s2 yet.
2022-09-19 14:28:09 +03:00
Roman Khavronenko
622bbedbe1
vmalert: always re-evaluate Annotations (#3119)
* vmalert: always re-evaluate Annotations

Previously, Annotations were evaluated only:
1. On alert creating.
2. On alert's value change.

This is premature optimization. It was assumed that since annotations
could contain only text with alert's labels or value - there is no need
in spending resources to re-compile Annotations.

Later, template function `query` was added, which can execute
arbitrary queries and return different results on every evaluation.
So if it was used in annotations, it would be executed only on init
or value change.

Another case when optimization caused an issue - annotations hot reload.
In this case, annotations of the active alert won't change even if Rule's
annotations were changed.

This fix enables Annotations re-evaluation on each iteration to resolve
issues above. It would have some impact on performance, but it is unlikely
it will be noticeable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add tp Changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-16 16:19:10 +02:00
Roman Khavronenko
3484673566
vmalert: add Troubleshooting section to docs (#3115)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-15 16:15:39 +02:00
Roman Khavronenko
9c95c81534
vmalert: print example of curl command for rule's state (#3112)
The change adds an example of `curl` command to the Rule's page.
The command is generated for each recorded state. It is supposed
user can just copy&execute the command to see what was returned
to vmalert.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-15 12:40:22 +02:00
Aliaksandr Valialkin
e7995375b5
docs: update -help output after explicit mentioning of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
4193af4571
docs/vmauth.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
3b2599c659
docs/vmalert.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
bbecd27557
docs: update docs with explicitly marked enterprise command-line flags for VictoriaMetrics and vmagent 2022-09-15 13:22:56 +03:00
Dmytro Kozlov
a9629cc32d
app/vmctl: add description about influx-skip-database-label flag (#3111) 2022-09-15 10:46:54 +02:00
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:41:09 +03:00
Roman Khavronenko
877940a131
vmalert: add experimental feature of storing Rule's evaluation state (#3106)
vmalert: add experimental feature of storing Rule's evaluation state

The new feature keeps last 20 state changes of each Rule
in memory. The state are available for view on the Rule's
view page. The page can be opened by clicking on `Details`
link next to Rule's name on the `/groups` page.

States change suppose to help in investigating cases when Rule
doesn't generate alerts or records.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 14:04:24 +02:00
Aliaksandr Valialkin
99bc18774c
app/vmselect/vmui: make vmui-update after 1304824201 2022-09-14 13:33:06 +03:00
Roman Khavronenko
efea51a9ee
bump Go version to 1.19.1 (#3108)
The reason is to cover vulnerability GO-2022-0969
Found in: net/http@go1.18.5
Fixed in: net/http@go1.19.1
More info: https://pkg.go.dev/vuln/GO-2022-0969

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 12:29:19 +02:00
Yury Molodov
1304824201
vmui: fix query params saving in URL (#3104) 2022-09-13 17:05:26 +02:00
Aliaksandr Valialkin
0ead64b6cf
app/vmalert: follow-up after 8441375da2
- Rename logDebug() to logDebugf() and pass format string together
  with format args directly to logDebugf(). This eliminates fmt.Sprintf()
  overhead at logDebug() call site when debugging is disabled.

- Format labels in debug message in Prometheus format, e.g. {label1="value1",...labelN="valueN"}

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025
2022-09-13 16:35:14 +03:00
Roman Khavronenko
8441375da2
vmalert: add debug mode for alerting rules (#3055)
* vmalert: add `debug` mode for alerting rules

Debug information includes alerts state changes and requests
sent to the datasource. Debug can be enabled only on rule's
level. It might be useful for debugging unexpected
behaviour of alerting rule.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmalert/alerting.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

* vmalert: go fmt

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-13 16:25:43 +03:00
Yury Molodov
1b41169415
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:55:11 +03:00
Yury Molodov
b5f4060520
fix: change columns for Top Queries (#3093) 2022-09-12 10:44:35 +03:00
Yury Molodov
9541ef2e9e
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:43:37 +03:00
Aliaksandr Valialkin
f81dfaf20d
deployment/docker: update Go builder for prod binaries from Go1.19.0 to Go1.19.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.1+label%3ACherryPickApproved
2022-09-08 18:27:10 +03:00
Aliaksandr Valialkin
ef7fdbb63c
docs/Single-server-VictoriaMetrics.md: add a note that cardinality explorer at cluster version of VictoriaMetrics may return lower than expected number of unique label values
See the corresponding comment in the code:

5a6e617b5e/app/vmselect/netstorage/netstorage.go (L1039-L1045)
2022-09-06 14:46:35 +03:00
Dmytro Kozlov
4415c71a2b
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049)
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
2022-09-06 13:25:59 +03:00
Aliaksandr Valialkin
651ace6ce4
docs/vmctl.md: make docs-sync after c5261d5f56
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:19:48 +03:00
Zakhar Bessarab
c5261d5f56
vmctl: implement support of chunking data for vm-native export (#3044)
vmctl: implement support of chunking data for vm-native export process

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 09:09:34 +02:00
Aliaksandr Valialkin
ae31b2363f
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:55:48 +03:00
匠心零度
50e2524bc2
api prometheus/api/v1/label/../values time not specified, (#3052)
modify default start values
2022-09-05 11:52:19 +03:00
Aliaksandr Valialkin
7dc632719d
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin
4076277cf0
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 19:46:27 +03:00
匠心零度
6d81584d2a
reduce unnecessary vmstorage query (#3031)
* reduce unnecessary vmstorage query

* reduce unnecessary vmstorage query

* rollback limit logic /api/v1/label/*
2022-08-30 12:36:54 +03:00
Aliaksandr Valialkin
d2e94ee91a
docs/CHANGELOG.md: document the 044d51b668 2022-08-30 09:42:19 +03:00