Aliaksandr Valialkin
b873b965af
app/vmagent/remotewrite: reduce memory usage when samples with big number of labels are sent to remote storage
2021-03-31 00:45:42 +03:00
Aliaksandr Valialkin
3fedf680f8
app/vmagent/remotewrite: remove superflouos code after 1b7dc1e5a5
2021-03-26 14:01:37 +02:00
Nikolay
15ff1f3a94
Adds blocks drop ( #1151 )
...
* adds blocks drop at 400 BadRequest status code
recieved from remote storage,
not expected that remote storage will be able to handle it on retry
* removes error logging for dropped blocks,
its expected error
2021-03-26 14:01:37 +02:00
Aliaksandr Valialkin
b91e18af0e
docs/vmagent.md: add an example for -remoteWrite.label
2021-03-25 17:59:30 +02:00
Aliaksandr Valialkin
2041008d64
app/vmagent: mention -remoteWrite.maxDiskUsagePerURL in the descriptio of -remoteWrite.tmpDataPath
flag
2021-03-23 16:38:41 +02:00
Aliaksandr Valialkin
5e77a939c2
all: make go vet
happy
2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
b997f4a418
all: make golangci-lint happy after the commit 6378205415
2021-03-17 00:24:31 +02:00
Aliaksandr Valialkin
8005ba26b9
lib/netutil: enable IPv6 UDP listening if -enableTCP6
command-line flag is passed to VictoriaMetrics
...
This is a follow-up for 18cfc4be7b
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
2021-03-17 00:19:30 +02:00
Aliaksandr Valialkin
323af49234
app/vmagent/remotewrite: clarify -remoteWrite.flushInterval
flag description
2021-03-01 11:51:08 +02:00
Aliaksandr Valialkin
b8dd5e8292
app/vmagent/remotewrite: fix rate limiting logic for -remoteWrite.url
2021-03-01 00:59:57 +02:00
Aliaksandr Valialkin
d0be1f6f49
app/vmagent: remove data race when applying rate limits to -remoteWrite.url
with multiple queues
2021-03-01 00:28:20 +02:00
Aliaksandr Valialkin
197ecca426
lib/promrelabel: add more optimizations for relabeling for common cases
2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin
72eef964d9
app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin
9c81429299
app/vmagent/remotewrite: cleanup after 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-17 21:43:48 +02:00
Nikolay
40973eda1c
adds pushback for fastqueue, ( #1075 )
...
during shutdown currently sending block was lost,
now its pushed back to fast queue and will be flushed on disk,
it may lead to data duplication.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-17 21:43:46 +02:00
Aliaksandr Valialkin
fdf9de98f8
app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values
...
This commit also adds --vm-round-digits command-line option to vmctl tool.
2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
4b324da947
all: consistently use timers from timerpool
2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
29bf531f7d
app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035
2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
4ee53c3961
all: use net.Dial
instead of fasthttp.Dial
, because fasthttp.Dial
limits the number of concurrent dials to 1000
2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin
49e800ba55
app/vmagent: add vmagent_remotewrite_blocks_sent_total
and vmagent_remotewrite_bytes_sent_total
metrics per each -remoteWrite.url
2020-12-15 20:41:08 +02:00
Nikolay
7064c4eb8e
adds new Array Flags ( #965 )
...
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url
* added backward compatibility testcases for ArrayDuration and ArrayBool
* fixes bool flag
* fixes test cases
2020-12-15 12:59:33 +02:00
Aliaksandr Valialkin
1a237c6903
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
7bdf07883b
app/{vmalert,vmagent}: skip empty values in -remoteWrite.label
and -label
lists
2020-12-08 14:54:02 +02:00
Aliaksandr Valialkin
b8083b7659
lib/promscrape: clean references to label name and label value strings after applying per-target relabeling
...
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin
887a3c317f
app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop
2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin
ca2e0f1e04
app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code
...
This makes consistent the behaviour with Prometheus.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin
5dca7bbe85
app/vmagent/remotewrite: do not show -remoteWrite.url
in logs if -remoteWrite.showURL
isn't set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin
8fc9b77496
app/vmagent: reduce memory usage when importing data via /api/v1/import
...
Previously vmagent could use big amounts of RAM when each ingested JSON line
contained many samples.
2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin
1c07d7bee7
app/vmagent: increase default value for -remoteWrite.queues
from 1 to 4, since it has been appeared that many users hit this limit
2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin
406f4fe445
app/vmagent: substitute -remoteWrite.url
with secret-url
value in logs, since it may contain sensitive info such as passwords or auth tokens
...
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin
6382e8081a
app/vmagent: allow setting multiple identical -remoteWrite.url
values
...
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin
a3cdef6b06
app/vmagent: properly flush big blocks of data
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741
Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin
de216bab41
app/vmagent: fix data race when accessing writeRequest.lastFlushTime
2020-09-03 12:12:09 +03:00
Aliaksandr Valialkin
6f9c1bc078
app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url
2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin
3b1ecac04b
app/vmagent: apply sane limits to -remoteWrite.queues
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
2020-08-30 21:25:51 +03:00
Roman Khavronenko
4b89da9463
lib/decimal: rename significant decimal digits
to significant figures
( #698 )
...
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures ) rounding
applied to all significant figures, not just decimal ones.
2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989
all: allow using KB
, MB
, GB
, KiB
, MiB
and GiB
suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
a930460236
app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin
baebe86844
app/vmagent/remotewrite: add missing resp.Body.Close()
after pushing data to remote storage
...
Missing body close could disable HTTP keep-alive connections.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin
d5057f6d04
app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage
...
Previously the request body was already consumed before the retry, so this led to the following error:
http: ContentLength=... with Body length 0
2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin
a3f48e395e
app/vmagent: add -remoteWrite.decimalPlaces
command-line flag, which may be used for reducing disk space usage on the remote storage
2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin
3b246aa569
app/vmagent: add -remoteWrite.proxyURL
command-line option
...
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin
86b54f3768
app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig
entries
2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin
c2b4b9138d
app/vmagent/remotewrite: return proper value from tssRelabelPool.New
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin
4353ff7ef1
app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig
options are set
...
Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race
and improper relabeling. Now each goroutine has its own copy of tss during relabeling.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-10 15:17:23 +03:00
Aliaksandr Valialkin
805a90f642
app/vmagent/remotewrite: typo fix in -remoteWrite.showURL
help message
2020-07-10 14:07:14 +03:00
Aliaksandr Valialkin
6ebac3ab63
app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig
command-line arg points to a file with a list of relabel_config
entries
...
See https://victoriametrics.github.io/#relabeling
2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
37aa4fe282
app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518
2020-05-30 14:37:02 +03:00
Aliaksandr Valialkin
a646131a33
app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent
2020-05-30 14:22:38 +03:00