Commit Graph

8760 Commits

Author SHA1 Message Date
hagen1778
67efc174a8
app/vmauth: properly initialize URLPrefix in tests
It is assumed that URLPrefix.busOriginal will be initialized
durin Unmarshal of the config. But in tests we set fields manually,
so this field never get initialized properly.

Fixes the error `panic: runtime error: integer divide by zero`
at `vmauth.getLeastLoadedBackendURL`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cb1e618a16)
2024-03-08 21:15:24 +01:00
hagen1778
86abbe5b67
app/vmctl: support TLS configuration for VictoriaMetrics destination
VictoriaMetrics destination is specified via `--vm-*` cmd-line flags
and is used in opentsdb, influx, prometheus, remote-read modes.

updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5426

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0b7ce70df4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-08 21:15:23 +01:00
hagen1778
b1fe25792c
app/vmctl: follow-up b9f7c3169a
* fix typos in flags description
* move the change to #tip section in changelog

b9f7c3169a
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 83a8c24281)
2024-03-08 21:15:09 +01:00
Khushi Jain
98c0909a12
app/vmctl : Provide TLS config options for vm native protocol (#5824)
Co-authored-by: Khushi Jain <khushi.jain@nokia.com>
(cherry picked from commit b9f7c3169a)
2024-03-08 21:15:09 +01:00
Aliaksandr Valialkin
b21fb355fa
docs/vmauth.md: update -help output after e08b91baafc95da090f75e9c29a27d8f62a2b76e 2024-03-07 01:37:48 +02:00
Aliaksandr Valialkin
b9788d5cf8
docs/CHANGELOG.md: typo fixes 2024-03-07 01:35:41 +02:00
Aliaksandr Valialkin
d9e3670627
lib/promauth: set the Host header to tlsServerName if itsn't empty
If tlsServerName isn't empty, then it is likely the https request is sent to IP instead of hostname.
In this case the request will fail, since Go automatically sets the Host header to the IP instead
of the desired hostname at tlsServerName. So set the Host header to tlsServerName if itsn't empty.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802
2024-03-07 01:23:40 +02:00
Aliaksandr Valialkin
e7e9721476
docs/vmauth.md: typo fixes after 7b2b980181 2024-03-07 01:08:40 +02:00
Aliaksandr Valialkin
93ef217857
app/vmauth: allow discovering backend ips behind shared hostname and spreading load among the discovered ips
This is done with the `discover_backend_ips` option at `user` and `url_map` level.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
2024-03-07 01:02:38 +02:00
Aliaksandr Valialkin
b2e182ca62
app/vmauth: add src_headers option at url_map, which allows routing incoming requests to different backends depending on request headers 2024-03-06 21:56:42 +02:00
Aliaksandr Valialkin
f5dd9b423e
vendor: run make vendor-update 2024-03-06 21:24:55 +02:00
Aliaksandr Valialkin
5bd4325135
app/vmauth: simplify configuration for src_query_args
Use the shorter form:

src_query_args:
- arg1=value1
- arg2=value2

instead of

src_query_args:
- name: arg1
  value: value2
- name: arg2
  value: value2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 21:24:53 +02:00
Aliaksandr Valialkin
1df7e5b69d
deployment/docker: update Go builder from Go1.21.7 to Go1.22.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved
2024-03-06 21:04:44 +02:00
Aliaksandr Valialkin
0929f255f0
docs/vmauth.md: mention that request query args can used for routing decisions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:58:10 +02:00
Aliaksandr Valialkin
87d59cdfa5
app/vmauth: add ability to route requests based on HTTP query args via src_query_args option
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:52:42 +02:00
Aliaksandr Valialkin
247d2f3118
app/vmauth: small code cleanup for working with auth tokens 2024-03-06 20:06:25 +02:00
Aliaksandr Valialkin
5817bf1c59
lib/streamaggr: add tests for keep_metric_names and drop_input_labels options 2024-03-06 20:06:23 +02:00
Aliaksandr Valialkin
03c65c9619
app/vmauth: use slices.Contains() instead of hasInt() 2024-03-06 17:36:02 +02:00
Aliaksandr Valialkin
1d7efc4c64
app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00
Aliaksandr Valialkin
2711732944
app/vmselect/prometheus: do not drop match[] filters if -search.ignoreExtraFiltersAtLabelsAPI flag is set
The `match[]` filter is mandatory at /api/v1/series, so it mustn't be dropped here.

There is no sense in dropping `match[]` filter together with `extra_label` and `extra_filters[]`
at /api/v1/labels and /api/v1/label/.../values if -search.ignoreExtraFiltersAtLabelsAPI commnad-line flag is set,
since:
- the `match[]` filter triggers slow path at these APIs;
- the `extra_label` and `extra_filters[]` filters narrow down the number of matched time series,
  so they improve performance comparing to the case when only `match[]` filter is left,
  while `extra_label` and `extra_filters[]` filters are dropped.

This is a follow-up for 0b7a23a91d
2024-03-06 13:57:20 +02:00
Daria Karavaieva
98b56476d7
redirect alias (#5934) 2024-03-06 13:54:48 +02:00
Yury Molodov
56e61bb54b
vmui: improve tracing styles (#5926)
Improved trace display for better visual separation of branches:
* Increased left padding for each element
* Added padding for the last element in the branch
2024-03-06 13:54:07 +02:00
hagen1778
a8d42ce4ed
lib/writeconcurrencylimiter: mention dependency on CPU cores for -maxConcurrentInserts flag
The change also removes misleading `default` value from README for `maxConcurrentInserts`
cmd-line flag.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-06 13:44:16 +02:00
Github Actions
cfce708eb9
Automatic update operator docs from VictoriaMetrics/operator@f028fdf (#5929) 2024-03-06 13:34:37 +02:00
hagen1778
764fc566ff
dashboards: add more context to cluster dashboard panels
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-06 13:34:10 +02:00
hagen1778
a4cdfaea28
docs: clarify deduplication is needed in multi-retention setup
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-06 13:33:40 +02:00
hagen1778
a364de3cfe
lib/writeconcurrencylimiter: mention dependency on CPU cores for -maxConcurrentInserts flag
The change also removes misleading `default` value from README for `maxConcurrentInserts`
cmd-line flag.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 73f5fb0f0c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-05 18:56:38 +01:00
Aliaksandr Valialkin
27b9e8ed3e
app/{vmagent,vminsert}: add -streamAggr.dropInputSamples command-line flag for dropping the specified labels from input samples before deduplication and streaming aggregation 2024-03-05 02:27:27 +02:00
hagen1778
b612c0536e
docs: re-fresh vmctl docs
* add recommendation about network bandiwdth between vmctl, source and destination
* use more relevant time series selector in examples - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5835

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-05 00:49:48 +02:00
Aliaksandr Valialkin
c38c45d71f
app/{vminsert,vmagent}: allow using -streamAggr.dedupInterval without -streamAggr.config
This allows performing online de-duplication of incoming samples
2024-03-05 00:47:23 +02:00
Aliaksandr Valialkin
4352544d61
lib/streamaggr: do not reset aggregation state after the aggregation took longer than the configured interval
It is better from user PoV preserving this state until the next flush
2024-03-04 20:03:45 +02:00
Aliaksandr Valialkin
9f81450b38
lib/streamaggr: add missing "s" suffix in the warning message when the de-duplication or aggregation couldnt be finished in a timely manner 2024-03-04 19:38:39 +02:00
Aliaksandr Valialkin
d10932bd99
lib/streamaggr: benchmark only flush routines in BenchmarkDedupAggrFlushSerial and BenchmarkAggregatorsFlushSerial 2024-03-04 19:13:50 +02:00
Aliaksandr Valialkin
36ee08cad4
Revert "lib/streamaggr: do not flush dedup shards in parallel"
This reverts commit eb40395a1c.

Reason for revert: it has been appeared that the performance gain on multiple CPU cores
wasn't visible because the benchmark was generating incorrect pushSample.key.

See a207e0bf687d65f5198207477248d70c69284296
2024-03-04 19:13:50 +02:00
Aliaksandr Valialkin
9728aaf5d9
lib/streamaggr: properly generate pushSample.key in benchmarks 2024-03-04 19:13:49 +02:00
Aliaksandr Valialkin
93a057e4e6
lib/streamaggr: reduce the number of pointers at "total" aggregation state
This should reduce load on GC when scanning heap objects.
2024-03-04 19:13:49 +02:00
Aliaksandr Valialkin
9e00d8ad60
lib/streamaggr: use multiple job label values in BenchmarkAggregatorsPush instead of single value
This should make the benchmark closer to production cases
2024-03-04 19:13:48 +02:00
Aliaksandr Valialkin
9773ad200e
lib/streamaggr: use multiple job labels in BenchmarkAggregatorsPush 2024-03-04 19:13:48 +02:00
Github Actions
b530f6bf73
Automatic update operator docs from VictoriaMetrics/operator@c8ff654 (#5918) 2024-03-04 19:12:59 +02:00
Aliaksandr Valialkin
482560a1f3
lib/streamaggr: do not flush dedup shards in parallel
This significantly increases CPU usage on systems with many CPU cores, while doesn't reduce flush latency too much
2024-03-04 17:01:42 +02:00
Aliaksandr Valialkin
d7252fce79
lib/streamaggr: reduce memory allocations when registering new series in deduplication and aggregation structs 2024-03-04 17:01:41 +02:00
Aliaksandr Valialkin
402dc14ec0
lib/streamaggr: make aggregate.runFlusher() more roubst and clear 2024-03-04 17:01:41 +02:00
Aliaksandr Valialkin
2ffef39bb3
lib/streamaggr: properly drop samples on the first incomplete interval
Previously samples were dropped on the first incomplete interval and the next complete interval.
Also make sure that the de-duplication is performed just before flushing the aggregate state.
This should help the case then dedup_interval = interval.
2024-03-04 17:01:40 +02:00
hagen1778
51745ec5ff
dashboards: update links in various panels
* use docs.victoriametrics.com instead of github docs
* add links to common terms used in VictoriaMetrics

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-04 17:00:54 +02:00
Aliaksandr Valialkin
c2dae136b3
lib/streamaggr: explicitly call resetSeries after flushSeries
This makes the code less fragile
2024-03-04 06:23:36 +02:00
Aliaksandr Valialkin
48a425898a
lib/streamaggr: enable time alignment for aggregate flushed to multiples of interval
For example, if `interval: 1m`, then data flush occurs at the end of every minute,
while `interval: 1h` leads to data flush at the end of every hour.

Add `no_align_flush_to_interval` option, which can be used for disabling the alignment.
2024-03-04 06:23:35 +02:00
Aliaksandr Valialkin
3ba9b2225e
docs/stream-aggregation.md: add troubleshooting section with solutions for common problems in streaming aggregation 2024-03-04 03:04:59 +02:00
Aliaksandr Valialkin
d80deaeaf4
lib/streamaggr: ignore the first sample in new time series during staleness_interval seconds after the stream aggregation start for total and increase outputs 2024-03-04 03:04:58 +02:00
Aliaksandr Valialkin
5e9cbfd4db
lib/streamaggr: flush dedup state and aggregation state in parallel on all the available CPU cores
This should reduce the time needed for aggregation state flush on systems with many CPU cores
2024-03-04 01:22:41 +02:00
Aliaksandr Valialkin
1e741ed6db
lib/streamaggr: add a benchmark for flushing dedup state 2024-03-04 01:22:40 +02:00