Commit Graph

3156 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
3f79e54a51
app/vmagent/remotewrite: follow-up for 166b97b8d0 and b6bd9a97a3
- Make the configuration more clear by accepting the list of ignored labels during sharding
  via a dedicated command-line flag - -remoteWrite.shardByURL.ignoreLabels.
  This prevents from overloading the meaning of -remoteWrite.shardByURL.labels command-line flag.

- Removed superfluous memory allocation per each processed sample if sharding by remote storage is enabled.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938
2024-04-03 00:50:48 +03:00
Hui Wang
fdb6eb1071
vmalert: fix sending alert messages (#6028)
* vmalert: fix sending alert messages
1. fix `endsAt` field in messages that send to alertmanager, previously rule with small interval could never be triggered;
2. fix behavior of `-rule.resendDelay`, before it could prevent sending firing message when rule state is volatile.

* docs: update changelog notes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-02 23:51:06 +03:00
Andrii Chubatiuk
9c92cc2759
vmauth: support other auth header names besides Authorization (#6009) 2024-04-02 18:41:00 +03:00
Andrii Chubatiuk
914b23f1e8
app/{vmagent,vminsert}: fixed firehose response (#6016) 2024-04-02 18:03:12 +03:00
Jiekun
1e0a079143
app/vmalert: respect batch size limit for remote write on shutdown (#6039)
During shutdown period of vmalert, remotewrite client retrieve all pending time series from buffer queue, compose them into 1 batch and execute remote write.

This final batch may exceed the limit of -remoteWrite.maxBatchSize, and be rejected by the receiver (gateway, vmcluster or others).

This changes ensures that even during shutdown vmalert won't exceed the max batch size limit for remote write
destination.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025

(cherry picked from commit 623d257faf)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-29 14:30:24 +01:00
hagen1778
36c1ca9949
app/vmagent: follow-up 166b97b8d0
* add tests for sharding function
* update flags description
* add changelog note

166b97b8d0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b6bd9a97a3)
2024-03-29 14:10:17 +01:00
Eugene Ma
744d513181
vmagent: support sharding by excluded labels (#5938)
To horizontally scale streaming aggregation, you might want to deploy a separate hashing tier
of vmagents that route to a separate aggregation tier. The hashing tier should shard by all labels
except the instance-level labels, to ensure the input metrics are routed correctly to the aggregator
instance responsible for those labels.
For this to achieve we introduce `remoteWrite.shardByURL.inverseLabels` flag to inverse logic of `remoteWrite.shardByURL.labels`

---------

Co-authored-by: Eugene Ma <eugene.ma@airbnb.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 166b97b8d0)
2024-03-29 14:10:15 +01:00
Roman Khavronenko
548bf31dd2
app/vmselect: make vmselect resilient to absence of cache folder (#5987)
vmselect uses a cache folder in file system for two purposes:
1. Storing rollup cache results on shutdown;
2. Storing temporary search results from vmstorage during query executions.

It could happen that cache folder is deleted accidentally by user, or by OS
during cleanup routines. This would cause vmselect to:
1. panic on /metrics call, because `MustGetFreeSpace` will fail;
2. return query error user, as it won't be able to store temporary search results.

The changes in this commit are the following:
1. Make `MustGetFreeSpace` to try re-creating the cache folder if it is missing;
2. Make vmselect to try re-creating the cache folder if it can't persist tmp search
results.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
(cherry picked from commit cb23685681)
2024-03-26 15:27:32 +01:00
rbizos
ab44787a70
adding AggregateSeriesLists graphite function (#5809)
* adding aggregate series list graphite function

adding also aliases for sum diff and multiply

* Adding tests for aggregateSeriesLists and aliases
2024-03-26 14:48:58 +01:00
Artem Navoiev
19a11c12d1
fix typo in kyiv city name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit b569fa0b2c)
2024-03-25 15:42:53 +01:00
Nikolay
60d64edd6e
app/vlselect: follow-up for 0514091948 (#6004)
removes println lines

(cherry picked from commit 43b5d8bc7a)
2024-03-25 15:42:53 +01:00
Alexander Marshalov
0b3744effa
[vmagent] added ingestion rate limiting with new flag -maxIngestionRate (#5900)
* [vmagent] added ingestion rate limiting with new flag `-maxIngestionRate`. This flag can be used to limit the number of samples ingested by vmagent per second. If the limit is exceeded, the ingestion rate will be throttled.

* fix changelog

* fix review comment

(cherry picked from commit 02bccd1eb9)
2024-03-25 15:42:52 +01:00
Yury Molodov
30778e0d0f
vmui: fix the _time filter insertion for all queries in VictoriaLogs UI #5920 (#5927)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920
(cherry picked from commit 46fd0ed693)
2024-03-25 15:42:33 +01:00
Dmytro Kozlov
9817ecdcc9
app/vmctl: break explore phase in vm-native mode by time intervals
When `--vm-native-step-interval` is specified, explore phase will be executed
within specified intervals. Discovered metric names will be associated with
time intervals at which they were discovered. This suppose to reduce number
of requests vmctl makes per metric name since it will skip time intervals
when metric name didn't exist.

This should also reduce probability of exceeding complexity limits
for number of selected series in one request during explore phase.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5369
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f8b91186a)
2024-03-25 15:42:24 +01:00
hagen1778
9fcd34287b
docs: follow-up for 70eaa06f08
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit bc79f7196d)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-25 15:33:11 +01:00
kbweave
1fc3b39b9c
app/vmselect: use GetDeadlineForLabelsAPI for LabelAPI requests (#5992)
(cherry picked from commit 70eaa06f08)
2024-03-25 15:31:21 +01:00
Aliaksandr Valialkin
111d0aa2bf
app/{vmagent,vminsert}: add an ability to ignore input samples outside the current aggregation interval for stream aggregation
See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples
2024-03-17 23:30:46 +02:00
hagen1778
efddda945e
app/vmctl: fix arguments order in httputils.TLSConfig
follow-up after 9d5bf5ba5d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-17 23:22:08 +02:00
Khushi Jain
147a63cafe
app/vmctl: fix the order of arguments in TLS config func (#5972) 2024-03-17 23:20:19 +02:00
Tien M. Nguyen
03fb97d42f
feat: include cluster info in alert CPUThrottlingHigh (#5956) 2024-03-17 20:46:15 +02:00
Hui Wang
349564fd82
vmalert: deprecate cmd-line flag -datasource.lookback (#5877)
* vmalert: deprecate cmd-line flag `-datasource.lookback`

* fix lint

* review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e80b44f19d)
2024-03-12 16:17:40 +01:00
hagen1778
67efc174a8
app/vmauth: properly initialize URLPrefix in tests
It is assumed that URLPrefix.busOriginal will be initialized
durin Unmarshal of the config. But in tests we set fields manually,
so this field never get initialized properly.

Fixes the error `panic: runtime error: integer divide by zero`
at `vmauth.getLeastLoadedBackendURL`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cb1e618a16)
2024-03-08 21:15:24 +01:00
hagen1778
86abbe5b67
app/vmctl: support TLS configuration for VictoriaMetrics destination
VictoriaMetrics destination is specified via `--vm-*` cmd-line flags
and is used in opentsdb, influx, prometheus, remote-read modes.

updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5426

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0b7ce70df4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-08 21:15:23 +01:00
hagen1778
b1fe25792c
app/vmctl: follow-up b9f7c3169a
* fix typos in flags description
* move the change to #tip section in changelog

b9f7c3169a
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 83a8c24281)
2024-03-08 21:15:09 +01:00
Khushi Jain
98c0909a12
app/vmctl : Provide TLS config options for vm native protocol (#5824)
Co-authored-by: Khushi Jain <khushi.jain@nokia.com>
(cherry picked from commit b9f7c3169a)
2024-03-08 21:15:09 +01:00
Aliaksandr Valialkin
93ef217857
app/vmauth: allow discovering backend ips behind shared hostname and spreading load among the discovered ips
This is done with the `discover_backend_ips` option at `user` and `url_map` level.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
2024-03-07 01:02:38 +02:00
Aliaksandr Valialkin
b2e182ca62
app/vmauth: add src_headers option at url_map, which allows routing incoming requests to different backends depending on request headers 2024-03-06 21:56:42 +02:00
Aliaksandr Valialkin
5bd4325135
app/vmauth: simplify configuration for src_query_args
Use the shorter form:

src_query_args:
- arg1=value1
- arg2=value2

instead of

src_query_args:
- name: arg1
  value: value2
- name: arg2
  value: value2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 21:24:53 +02:00
Aliaksandr Valialkin
1df7e5b69d
deployment/docker: update Go builder from Go1.21.7 to Go1.22.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved
2024-03-06 21:04:44 +02:00
Aliaksandr Valialkin
87d59cdfa5
app/vmauth: add ability to route requests based on HTTP query args via src_query_args option
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:52:42 +02:00
Aliaksandr Valialkin
247d2f3118
app/vmauth: small code cleanup for working with auth tokens 2024-03-06 20:06:25 +02:00
Aliaksandr Valialkin
03c65c9619
app/vmauth: use slices.Contains() instead of hasInt() 2024-03-06 17:36:02 +02:00
Aliaksandr Valialkin
1d7efc4c64
app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00
Aliaksandr Valialkin
2711732944
app/vmselect/prometheus: do not drop match[] filters if -search.ignoreExtraFiltersAtLabelsAPI flag is set
The `match[]` filter is mandatory at /api/v1/series, so it mustn't be dropped here.

There is no sense in dropping `match[]` filter together with `extra_label` and `extra_filters[]`
at /api/v1/labels and /api/v1/label/.../values if -search.ignoreExtraFiltersAtLabelsAPI commnad-line flag is set,
since:
- the `match[]` filter triggers slow path at these APIs;
- the `extra_label` and `extra_filters[]` filters narrow down the number of matched time series,
  so they improve performance comparing to the case when only `match[]` filter is left,
  while `extra_label` and `extra_filters[]` filters are dropped.

This is a follow-up for 0b7a23a91d
2024-03-06 13:57:20 +02:00
Yury Molodov
56e61bb54b
vmui: improve tracing styles (#5926)
Improved trace display for better visual separation of branches:
* Increased left padding for each element
* Added padding for the last element in the branch
2024-03-06 13:54:07 +02:00
Aliaksandr Valialkin
27b9e8ed3e
app/{vmagent,vminsert}: add -streamAggr.dropInputSamples command-line flag for dropping the specified labels from input samples before deduplication and streaming aggregation 2024-03-05 02:27:27 +02:00
Aliaksandr Valialkin
c38c45d71f
app/{vminsert,vmagent}: allow using -streamAggr.dedupInterval without -streamAggr.config
This allows performing online de-duplication of incoming samples
2024-03-05 00:47:23 +02:00
Aliaksandr Valialkin
48a425898a
lib/streamaggr: enable time alignment for aggregate flushed to multiples of interval
For example, if `interval: 1m`, then data flush occurs at the end of every minute,
while `interval: 1h` leads to data flush at the end of every hour.

Add `no_align_flush_to_interval` option, which can be used for disabling the alignment.
2024-03-04 06:23:35 +02:00
Aliaksandr Valialkin
334b90598e
app/vlselect/vmui: run make vmui-logs-update after c51031dd70
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5799
2024-03-01 01:33:37 +02:00
Yury Molodov
e00d313333
vmui: add field for log entries limit (#5799)
* vmui: add field for log entries limit (#5674)

* vmui: refactor useFetchLogs

* vmui: fix log query encoding

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-03-01 01:33:35 +02:00
Aliaksandr Valialkin
d84cdf98da
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after e130f29659
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152
2024-03-01 00:51:49 +02:00
Yury Molodov
c71c2b37c5
vmui: add gap display option for charts #5152 (#5862) 2024-03-01 00:51:48 +02:00
Aliaksandr Valialkin
46d92fae3d
app/vmselect/prometheus: ignore match[] additionally to extra_filters[] and extra_label if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set
The match[] at /api/v1/labels and /api/v1/label/.../values also may lead to slow requests and
high resource usage if it matches big number of time series. So it must be igrnored if -search.ignoreExtraFiltersAtLabelsAPI
command-line flag is set.

This is a follow-up for fab02faa3f
2024-03-01 00:18:08 +02:00
Aliaksandr Valialkin
6aea1e5093
app/{vmagent,vminsert}: follow-up for 434a5803e7
Document the /opentelemetry/v1/metrics endpoint instead of /opentelemetry/api/v1/push,
since the /v1/metrics suffix is hardcoded in OpenTelemetry protocol specification.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5871
2024-02-29 18:05:12 +02:00
Nikolay
abb773120d
app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path (#5871)
* app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path
it must fix compatibility with opentemetry-collector [spec](https://opentelemetry.io/docs/specs/otlp/\#otlphttp-request)
this suffix is hard-coded and cannot be changed with collector configuration

* Apply suggestions from code review

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 17:59:57 +02:00
Aliaksandr Valialkin
fed04e7f25
app/vmselect/promql: use unsafe.Slice instead of deprecated reflect.SliceHeader 2024-02-29 17:51:16 +02:00
Aliaksandr Valialkin
319d21eddf
app/vmselect/netstorage: usae unsafe.SliceData instead of deprecated reflect.SliceHeader 2024-02-29 17:38:14 +02:00
Hui Wang
25e454be4c
metricsql: fix label_join() when dst_label is equal to one of the `… (#5886)
* metricsql: fix label_join() when `dst_label` is equal to one of the `src_label`

* Update app/vmselect/promql/transform.go

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 16:02:48 +02:00
Aliaksandr Valialkin
9d5a385973
app/{vmalert,vmctl}: consistently use http.NewRequestWithContext() instead of http.NewRequest() + req.WithContext() 2024-02-29 15:25:36 +02:00
Aliaksandr Valialkin
7832d0800e
app/{vminsert,vmagent}: follow-up after 67a55b89a4
- Document the ability to read OpenTelemetry data from Amazon Firehose at docs/CHANGELOG.md

- Simplify parsing Firehose data. There is no need in trying to optimize the parsing with fastjson
  and byte slice tricks, since OpenTelemetry protocol is really slooow because of over-engineering.
  It is better to write clear code for better maintanability in the future.

- Move Firehose parser from /lib/protoparser/firehose to lib/protoparser/opentelemetry/firehose,
  since it is used only by opentelemetry parser.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5893
2024-02-29 14:47:20 +02:00