Commit Graph

2401 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
c6382d9538
app/vmctl/README.md: remove trailing space after cc5b916237 2023-03-08 01:38:27 -08:00
Aliaksandr Valialkin
e91fe74618
all: typo fixes of the same type as in the d056be710b 2023-03-08 01:37:28 -08:00
Aliaksandr Valialkin
b60594a548
app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c
- Document the fix
- Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient()
  This simplifies the fix to the following diff:

diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go
index 099899c19..70b904af4 100644
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
        }
        c.sendBlock = c.sendBlockHTTP

-       return c
-}
-
-func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        useVMProto := forceVMProto.GetOptionalArg(argIdx)
        usePromProto := forcePromProto.GetOptionalArg(argIdx)
        if useVMProto && usePromProto {
@@ -173,6 +169,10 @@ func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        }
        c.useVMProto = useVMProto

+       return c
+}
+
+func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
2023-03-08 01:35:31 -08:00
Dmytro Kozlov
2cf6797a24
app/vmagent: fix panic if auth config not defined (#530) 2023-03-08 01:35:04 -08:00
Nikolay
88f10d24a0
lib/netutil: fixes panic at proxy protocol (#3905)
it may occur if non proxy protocol message received by tcp server.
Listener Accept method must return only non-recoverable errors.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:33:01 -08:00
Yury Molodov
3e3e23a483
vmui: fix display of selected value in the selector (#3919)
vmui: fix selected value in dropdowns for Explore page
2023-03-08 01:13:54 -08:00
Roman Khavronenko
fa3b2bd205
app/vmalert: do not wait for group start on removal (#3891)
Each group in vmalert starts with an artifical delay to avoid
thundering herd problem. For some groups with high evaluation
intervals, the delay could be significant.
If during this delay user will remove the group from the config
and hot-reload it - vmalert will have to wait until the delay
ends. This results into slow config reloading and UI hang.

The change moves the start-delay logic back to the group's
`start` method. Now, group can immediately exit from the
delay when `group.close()` method is called.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 01:10:41 -08:00
Dmytro Kozlov
d59a20c0a7
docs: follow up after 4b136abff8 (#3918)
docs: follow up after 4b136abff8
2023-03-08 01:08:07 -08:00
Gowtam Lal
0efdf47571
app/vmctl: Add ability to set headers for vm-native HTTP requests. (#3906)
app/vmctl: Add ability to set headers for vm-native HTTP requests
2023-03-08 01:06:32 -08:00
Roman Khavronenko
b37ae04ae0
docs: follow-up e781e22c9c (#3902)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 00:51:34 -08:00
Yury Molodov
c383c9b864
vmui: show query error (#3890)
* add links support with old query params
* show error after execute query
2023-03-08 00:49:36 -08:00
Dmytro Kozlov
ff0e63ef0d
app/vmctl: add backoff retries to native protocol (#3859)
app/vmctl: vm-native - split migration on per-metric basis

`vm-native` mode now splits the migration process on per-metric basis. 
This allows to migrate metrics one-by-one according to the specified filter. 
This change allows to retry export/import requests for a specific metric and provides a better 
understanding of the migration progress.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 00:24:08 -08:00
Roman Khavronenko
b176247e16
vmalert: cancel in-flight requests on group's update or close (#3886)
When group's update() or close() method is called, the group
still need to wait for its current evaluation to finish.
Sometimes, evaluation could take a significant amount of time
which slows configuration update or vmalert's graceful shutdown.

The change interrupts current evaluation in order to speed up
the graceful shutdown or config update procedures.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 00:10:11 -08:00
Dmytro Kozlov
08df754903
app/vmctl: Increase http request timeout made by remote read client, add importing tips (#3879)
app/vmctl: Increase default http request timeout made by remote read client
2023-03-07 23:35:25 -08:00
Aliaksandr Valialkin
086a4b4fca
lib/bytesutil: add -internStringDisableCache and -internStringCacheExpireDuration command-line flags
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872
2023-02-27 14:18:02 -08:00
Aliaksandr Valialkin
1ad0d22e80
lib/storage: follow-up for 39cdc546dd
- Use flag.Duration instead of flagutil.Duration for -snapshotCreateTimeout,
  since the flagutil.Duration is intended mostly for big durations, e.g. days, months and years,
  while the -snapshotCreateTimeout is usually smaller than one hour.
- Add links to https://docs.victoriametrics.com/#how-to-work-with-snapshots in docs/CHANGELOG.md,
  so readers could easily find the corresponding docs when reading the changelog.
- Properly remove all the created directories on unsuccessful attempt to create
  snapshot in Storage.CreateSnapshot().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
2023-02-27 13:11:10 -08:00
Zakhar Bessarab
26682e369e
lib/storage: enhancements for snapshots process (#3873)
* lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858)

* lib/{mergeset,storage}: add timeout configuration for snapshots creation, remove incomplete snapshots from storage

* docs: fix formatting

* app/vmstorage: add metrics to track status of snapshots

* app/vmstorage: use `vm_http_requests_total` metric for snapshot endpoints metrics, rename new flag to make name more clear

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update flag name in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: reflect new metrics names change in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 13:11:06 -08:00
Aliaksandr Valialkin
d3605ad072
app/vmselect/promql: fix panic when calculating aggr_func(rollup*())
The panic has been introduced in dac21d874b
2023-02-27 11:48:38 -08:00
Zakhar Bessarab
215206bded
app/vmgateway: add new flag doc
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:37 -08:00
Zakhar Bessarab
08a61d5c13
app/vmgateway: fix typo in docs
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:11 -08:00
Zakhar Bessarab
1b4871a9da
app/vmgateway: add OpenID discovery of JWKS endpoints
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:17:44 -08:00
Aliaksandr Valialkin
d9581cf3f3
app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol 2023-02-27 11:04:11 -08:00
Dmytro Kozlov
65d9e281d6
app/vmctl: skip series if measurement not found (#3869)
app/vmctl: skip measurements with no fields for influxdb mode
2023-02-27 10:35:13 -08:00
Dmytro Kozlov
afa3678b01
app/vmctl: enable version flag (#3868) 2023-02-27 10:34:26 -08:00
Aliaksandr Valialkin
bbd5914eb1
all: add makefile rules for GOARCH=s390x for all the VictoriaMetrics components
This is a follow-up for 007530f882
2023-02-26 12:38:48 -08:00
Aliaksandr Valialkin
18dd0d1dbf
.golangci.yml: properly enable revive linter and fix all the warnings it detects 2023-02-26 12:19:58 -08:00
Aliaksandr Valialkin
1e156ac3c3
app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-26 12:19:53 -08:00
Roman Khavronenko
66d0b45651
vmselect/promql: check for deadline in count_values fn (#3806)
* vmselect/promql: check for deadline in `count_values` fn

`count_values` could be very slow during the data processing.
Checking for deadline between iterations supposed to reduce
probability of exceeding `search.maxQueryDuration`.

The change also adds a new trace record, which captures the time
spent in aggregation function. Before that, the trace for aggr funcs
could be confusing since it doesn't account for all the places where
time was spent.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 17:10:38 -08:00
Roman Khavronenko
79eb33556e
metricsql: support optional 2nd argument for rollup functions (#3841)
* metricsql: support optional 2nd argument for rollup functions

Support optional 2nd argument `min`, `max` or `avg` for rollup functions:
 * rollup
 * rollup_delta
 * rollup_deriv
 * rollup_increase
 * rollup_rate
 * rollup_scrape_interval

 If second argument is passed, then rollup function will return only the selected aggregation type.
 This change can be useful for situations where only one type of rollup calculation is needed.
 For example, `rollup_rate(requests_total[5m], "max")`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 13:48:30 -08:00
Aliaksandr Valialkin
8efa9159cf
app/vmselect/promql: measure the time required for calculating the aggregate function from the prepared source time series 2023-02-23 20:06:02 -08:00
Aliaksandr Valialkin
dce8afa4c4
app/vmselect/vmui: make vmui-update after d4fc0ed874 2023-02-23 19:26:08 -08:00
Yury Molodov
57c257810d
vmui: improve mobile ui (#3848)
* feat: improve mobile ui

* feat: improve mobile ui

* fix: change style server url

* fix: improve ExploreMetrics mobile

* fix: display global settings on all pages
2023-02-23 19:26:01 -08:00
Aliaksandr Valialkin
3563c6882e
docs: update --help descriptions after recent changes 2023-02-23 19:02:32 -08:00
Aliaksandr Valialkin
6369c88a68
app/vmselect: add -search.logQueryMemoryUsage command-line flag for logging queries, which take big amounts of memory
Thanks to @michal-kralik for initial attempts for this feature:

- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3651
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3715

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553
2023-02-23 18:52:44 -08:00
Aliaksandr Valialkin
f579cac297
app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol
Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag,
which can be used for forcing Prometheus remote write protocol in cases when the remote storage
supports VictoriaMetrics remote write protocol.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-23 17:38:47 -08:00
Aliaksandr Valialkin
0c60e4a30a
all: consistently use http.Method{Get,Post,Put} across the codebase
This is a follow-up after 9dec3c8f80
2023-02-22 19:01:09 -08:00
my-git9
7d86c5c94a
chore: Use http constants to replace numbers (#3846)
Signed-off-by: xin.li <xin.li@daocloud.io>
2023-02-22 18:59:32 -08:00
Alexander Marshalov
9bb9bd266b
fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-22 18:41:29 -08:00
Yury Molodov
ecdbcb16ed
fix: change query params update (#3860) 2023-02-22 18:26:33 -08:00
Aliaksandr Valialkin
1b70238dca
lib/promscrape/discovery/kuma: follow-up for 317fef95f9
- Do not generate __meta_server label, since it is unavailable in Prometheus.
- Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md,
  so users could click it and read the docs without the need to search the corresponding docs.
- Remove kumaTarget struct, since it is easier generating labels for discovered targets
  directly from the response returned by Kuma. This simplifies the code.
- Store the generated labels for discovered targets inside atomic.Value. This allows reading them
  from concurrent goroutines without the need to use mutex.
- Use synchronouse requests to Kuma instead of long polling, since there is a little sense
  in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval.
- Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used.
- Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus.
- Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability.
- Remove unused fields from discoveryRequest and discoveryResponse.
- Update tests.
- Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config.
- Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback,
  since these are public types.

Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label,
since usually this label is set by the Prometheus itself to __address__ after the relabeling phase.
See https://www.robustperception.io/life-of-a-label/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389

See https://github.com/prometheus/prometheus/issues/7919
and https://github.com/prometheus/prometheus/pull/8844
as a reference implementation in Prometheus
2023-02-22 17:50:54 -08:00
Aliaksandr Valialkin
b3aed1aebc
app/vmagent/remotewrite: removed unneeded code in testPushWriteRequest after 57801660ab 2023-02-22 17:50:03 -08:00
Alexander Marshalov
173643a771
add kuma_sd_config for Kuma Control Plane targets discovery (#3389) (#3840) 2023-02-22 17:41:43 -08:00
Dmytro Kozlov
88d2a6c3a3
app/vmctl: add retry backoff policy (#3844)
app/vmctl: move retries logic into a separate pkg
2023-02-22 12:17:21 -08:00
Alexander Marshalov
29fa78a310
Fix TestPushWriteRequest for remotewriteprotocol for pure-test (with native zstd) (#3850)
app/vmagent: fix TestPushWriteRequest for pure-test (with native zstd)

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-02-22 12:16:29 -08:00
Artem Navoiev
522179fbde
docs: fix typo OpentTSDB -> OpenTSDB (#3854)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 12:11:48 -08:00
Aliaksandr Valialkin
ff8c57a964
app/vmselect: allow zero value for -search.latencyOffset command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2023-02-21 18:07:27 -08:00
Roman Khavronenko
a29c1d3a02
docs: mention rules replay blogpost in vmalert docs (#3851)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-21 17:44:32 -08:00
Zakhar Bessarab
8a35dc608f
vmgateway: add support of JWKS endpoint usage for JWT keys verification (#521) 2023-02-20 19:23:00 -08:00
Roman Khavronenko
fd139b463b
docs: update vmalert docs (#3843)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-20 19:13:00 -08:00
Aliaksandr Valialkin
fc5ea31a38
docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol
The 10x savings are reproduced only on artificial data.
The savings on production data are usually in the range 2x-4x.
2023-02-20 19:11:11 -08:00