Commit Graph

9167 Commits

Author SHA1 Message Date
Roman Khavronenko
ff73b66182
app/vmbackup: introduce new flag type URL (#6152)
The new flag type is supposed to be used for specifying URL values which
could contain sensitive information such as auth tokens in GET params or
HTTP basic authentication.

The URL flag also allows loading its value from files if `file://`
prefix is specified. As example, the new flag type was used in
app/vmbackup as it requires specifying `authKey` param for making the
snapshot.

See related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5973

Thanks to @wasim-nihal for initial implementation
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6060

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 029060af60)
2024-04-24 17:08:24 +02:00
Github Actions
ff0a1eb3e6
Automatic update operator docs from VictoriaMetrics/operator@6e1a876 (#6179)
(cherry picked from commit a14acd4fcc)
2024-04-24 17:08:21 +02:00
Artem Navoiev
4f07d32548
docs: vmalert remove new lines in configuration section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 592a9fe9f2)
2024-04-24 17:08:19 +02:00
Artem Navoiev
f22cfdc616
docs: vmalert remove new lines in configuration section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit aeba3f39db)
2024-04-24 17:08:18 +02:00
hagen1778
d4b56d467f
dashboards: show max number of active merges instead of cumulative
The cumulative number of active merges could be red herring
as it its value depends on the number of vmstorages.
For example, vmstorage could be added or removed and this will affect
the panel.
Or, each vmstorage could start a merging process (i.e. for downsampling)
and visiually it could look like a massive change.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 035de57e5e)
2024-04-24 17:08:15 +02:00
hagen1778
fcff6cc12a
docs/changelog: mention icnreased CPU usage in 1.100.1 release for ENT distributions
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 22cfab1ea2)
2024-04-24 17:08:13 +02:00
hagen1778
57c841669c
app/vmagent: mention corner case with dangling queues and identical URLs
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6140

We don't cover this corner case as it has low chance for reproduction.
Precisely, the requirements are following:
1. vmagent need to be configured with multiple identical `remoteWrite.url` flags;
2. At least one of the persistent queues need to be non-empty, which already
signalizes about issues with setup;
3. vmagent need to be restarted with removing of one of `remoteWrite.url` flags.

We do not document this case in vmagent.md as it seems to be a rare corner case
and its explanation will require too much of explanation and confuse users.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4251292708)
2024-04-23 14:52:35 +02:00
Github Actions
f2897fb859
Automatic update operator docs from VictoriaMetrics/operator@e343a8e (#6168)
(cherry picked from commit 301bd387d4)
2024-04-23 14:52:33 +02:00
Andrii Chubatiuk
29f9b8ef5d
deployment/docker: allow cross-platform building on arm64 platform (#6158)
Added x86_64 libraries to allow building cross-platform images on arm64

(cherry picked from commit 7958f38864)
2024-04-23 14:52:30 +02:00
Denys Holius
3c8aba3029
deployment/docker/victorialogs/fluentbit-docker/docker-compose.yml: update fluentbit version from v2.1.4 to v3.0.2 (#6120)
see also https://fluentbit.io/announcements/v3.0.0/

(cherry picked from commit bde4693a90)
2024-04-23 14:52:28 +02:00
Roman Khavronenko
2566e19306
app/vmalert: fix links with anchors in vmalert's UI (#6146)
Starting from v1.99.0 vmalert could ignore anchors pointing to specific
rule groups if `search` param was present in URL.
This change makes anchors compatible with `search` param in UI.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f487c7090)
2024-04-22 15:05:23 +02:00
Github Actions
d1ee860d46
Automatic update operator docs from VictoriaMetrics/operator@7e2ba6f (#6165)
(cherry picked from commit 6dbb4c1671)
2024-04-22 15:05:17 +02:00
Denys Holius
cad2cb2b56
docs: removed code-style highlighting for commanad-line flags of VM components (#6147)
Using `sh` or `console` formatting doesn't do word-breaking on render. This makes flags description
harder to read, as users need to scroll the web page horizontally.
Removing the formatting renders the description with normal word-breaking.

(cherry picked from commit 9bedbcfa2f)
2024-04-22 15:04:49 +02:00
hagen1778
342290275e
app/streamaggr: follow-up after c0e4ccb7b5
* rm vmagent mentions from vminsert flags
* improve documentation wording, add links to related sections
* mention `ignore_first_intervals` in the stream aggr options
* update flags description
* add basic test for config parsing validation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit bae3874e6a)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-22 14:39:23 +02:00
Andrii Chubatiuk
131367fb59
lib/streamaggr: add option to ignore first N aggregation intervals (#6137)
Stream aggregation may yield inaccurate results if it processes incomplete data.
This issue can arise when data is sourced from clients that maintain a queue of unsent data, such as Prometheus or vmagent.
 If the queue isn't fully cleared within the aggregation interval, only a portion of the time series may be included in that period, leading to distorted calculations.
To mitigate this we add an option to ignore first N aggregation intervals. It is expected, that client queues
will be cleared during the time while aggregation ignores first N intervals and all subsequent aggregations
will be correct.

(cherry picked from commit c0e4ccb7b5)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-22 14:34:36 +02:00
Aliaksandr Valialkin
ff122798b1
docs/MetricsQL.md: add links to raw samples chapter at https://docs.victoriametrics.com/keyconcepts/ 2024-04-21 00:22:03 +02:00
Aliaksandr Valialkin
946757fa5e
docs/CONTRIBUTING.md: small re-wording 2024-04-21 00:08:02 +02:00
Aliaksandr Valialkin
c254a889f7
docs: remove duplicate information regarding contribution guidelines to VictoriaMetrics
Substitute duplicate information with the link to https://docs.victoriametrics.com/contributing/ across all the docs.
2024-04-21 00:01:33 +02:00
Aliaksandr Valialkin
aa496d8827
docs/CONTRIBUTING.md: more formatting fixes and clarifications 2024-04-20 23:49:13 +02:00
Aliaksandr Valialkin
c87e383634
.github: move pull_request_template.md from .github/PULL_REQUEST_TEMPLATE/ to .github/
I couldn't figure out how to show the provided pull request template to users
according to https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository ,
I let's try 'global pull request for the whole repository' approach.
2024-04-20 23:43:06 +02:00
Aliaksandr Valialkin
59648c880a
docs/CONTRIBUTING.md: small corrections after a402847eb6 2024-04-20 23:22:04 +02:00
Aliaksandr Valialkin
b16dbab137
Move CONTRIBUTING.md to docs/
It is better from visibility PoV if the CONTRIBUTING.md file is visible at https://docs.victoriametrics.com/contributing/ .
This page can be indexed by search engines and searched by our users later.
It is also easier to provide a link to this page now comparing to the old link, which is much harder to remember:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md

While at it, syncrhonize docs/CONTRIBUTING.md with .github/PULL_REQUEST_TEMPLATE/pull_request_template.md ,
so they do not contain duplicate information, which can be outdated over time. Now all the relevant information
is located at docs/CONTRIBUTING.md, while .github/PULL_REQUEST_TEMPLATE/pull_request_template.md refers this doc.

This is a follow-up for c006db1798

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6040
2024-04-20 23:12:53 +02:00
Aliaksandr Valialkin
4318f34644
lib/protoparser: substitute hybrid channel-based pools with plain sync.Pool
Using plain sync.Pool simplifies the code without increasing memory usage and CPU usage.
So it is better to use plain sync.Pool from readability and maintainability PoV.

This is a follow-up for 8942f290eb
2024-04-20 22:02:39 +02:00
Aliaksandr Valialkin
c146b24196
app/vminsert: replace hybrid sync.Pool+channel-based pool scheme for poolCtx with plain sync.Pool
This simplifies the code, while doesn't increase memory usage under low and high data ingestion rate.

This is a follow-up for 1decbcf6eb
2024-04-20 21:46:11 +02:00
Aliaksandr Valialkin
71d0020c2f
app/vminsert/influx: replace hybrid channel-based pool+sync.Pool with plain sync.Pool for pushCtx
The memory usage for plain sync.Pool doesn't increase comparing to the memory usage for the hybrid scheme,
so it is better to use plain sync.Pool in order to simplify the code and make it more readable and maintainable.

This is a follow-up for c22da2f917
2024-04-20 21:41:18 +02:00
Aliaksandr Valialkin
a249ab96b4
app/vmagent/influx: replace hybrid channel-based pool + sync.Pool with plain sync.Pool for pushCtx
Data ingestion benchmark doesn't show memory usage difference between two approaches,
so let's use simpler approach in order to improve code readability and maintainability.

This is a follow-up for 77c597738c
2024-04-20 21:38:25 +02:00
Aliaksandr Valialkin
d6da68ee90
app/vmagent/common: use plain sync.Pool instead of a mix of sync.Pool with channel-based pool for PushCtx
This scheme was used for reducing memory usage when vmagent runs on a machine with big number of CPU cores
and the ingestion rate isn't too big. The scheme with channel-based pool could reduce memory usage,
since it minimizes the number of PushCtx structs in the pool in this case.

Performance tests didn't reveal significant difference in memory usage under both low and high ingestion rate
between plain sync.Pool and the current hybrid scheme, so replace the scheme with plain sync.Pool in order
to simplify the code.
2024-04-20 21:31:14 +02:00
Aliaksandr Valialkin
9004bc098e
all: use clear() built-in Go function for clearing []prompbmarshal.TimeSeries and []prompbmarshal.Label slices
This makes the code a bit clear.
2024-04-20 21:00:24 +02:00
Aliaksandr Valialkin
498fe1cfa5
app/vminsert/common: remove obsolete optimization for reducing memory usage for InsertCtx pool
This optimization is no longer needed according to benchmarks with ingestion rate.

This simplifies the code a bit.
2024-04-20 20:51:38 +02:00
Bartosz Fenski
b36661539f
typo (#6159)
s/infromation/information/
2024-04-20 20:49:16 +02:00
Github Actions
ac7746d7fa
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@5b8a0ba (#6156) 2024-04-20 20:48:44 +02:00
Aliaksandr Valialkin
fba3c10ed1
app/vmselect/promql: add support for matching against multiple numeric constants via q == (c1,...,cN) and q != (c1,...,cN) syntax 2024-04-19 17:57:09 +02:00
Aliaksandr Valialkin
5504b6f2bd
docs/CHANGELOG.md: document v1.97.4 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.4
2024-04-19 13:10:37 +02:00
Aliaksandr Valialkin
a40a900b43
docs/CHANGELOG.md: document v1.93.14 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14
2024-04-19 12:52:38 +02:00
Aliaksandr Valialkin
a674029111
docs/LTS-releases.md: change the latest v1.93.x LTS release from v1.93.13 to v1.93.14
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14
2024-04-19 12:52:36 +02:00
Aliaksandr Valialkin
5b29be1f4d
app/vmagent/remotewrite: add support for replication additionally to sharding when both -remoteWrite.shardByURL and -remoteWrite.shardByURLReplicas=RF command-line flags are set
This allows setting up data replication among failure domains if the replication factor is smaller than the number of failure domains.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054

See https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages
2024-04-19 11:37:04 +02:00
Github Actions
0fde6f4b8f
Automatic update operator docs from VictoriaMetrics/operator@3d7bcba (#6149) 2024-04-19 11:36:38 +02:00
Github Actions
66f3f90f74
Automatic update operator docs from VictoriaMetrics/operator@781dbba (#6143) 2024-04-19 11:35:58 +02:00
Github Actions
a0b1c95787
Automatic update operator docs from VictoriaMetrics/operator@731de81 (#6139)
Co-authored-by: Alexander Marshalov <_@marshalov.org>
2024-04-19 11:35:21 +02:00
Github Actions
3cb3a9c950
Automatic update operator docs from VictoriaMetrics/operator@7f6c6d8 (#6138) 2024-04-19 11:34:50 +02:00
Hui Wang
e0d47ab6af
vmalert: avoid blocking APIs when alerting rule uses template functio… (#6129)
* vmalert: avoid blocking APIs when alerting rule uses template function `query`

* app/vmalert: small refactoring

* simplify labels and templates expanding
* simplify `newAlert` interface
* fix `TestGroupStart` which mistakenly skipped annotations
and response labels check

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* reduce alerts lock time when restore

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-04-19 11:30:40 +02:00
Roman Khavronenko
95b0f82c9b
app/vmalert: make TestGroupStart more reliable (#6130)
There was a sleep statement in the test, waiting for Group
to perform a couple of evaluation. But looks like
it worked unreliable for some CI tests like the one below
https://github.com/VictoriaMetrics/VictoriaMetrics/actions/runs/8718213844/job/23915007958?pr=6115

This commit changes the sleep statement on a function that
waits for a specific number of evaluations. It should make this
test faster in general case, and more reliable for slow environemnts.
2024-04-19 11:28:30 +02:00
Aliaksandr Valialkin
f3967737b0
docs/Cluster-VictoriaMetrics.md: typo fix after dadeb6620a: folling -> following 2024-04-19 00:24:52 +02:00
Aliaksandr Valialkin
90794e84bc
app/vmselect/netstorage: add support for the ability to set cross-group replication factor at vmselect
The cross-group replication factor can be set via `-globalReplicationFactor` command-line flag at vmselect.
In this case vmselect continues returning full responses if up to globalReplicationFactor-1 groups are unavailable.

See https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect for details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054
2024-04-19 00:16:04 +02:00
Aliaksandr Valialkin
e138f4827e
lib/storage: search for all the values for the given label before applying filters and limits
It is incorrect applying the limit on the number of values to search without applying filters,
since the returned subset of label values may miss the label values matching the given filters.

This is a follow-up for 66630c7960
2024-04-18 20:40:19 +02:00
Aliaksandr Valialkin
b523893a67
app/{vlselect,vmselect}: run make vmui-update vmui-logs-update 2024-04-18 17:37:28 +02:00
Aliaksandr Valialkin
e2bc49534b
all: replace old https://docs.victoriametrics.com/PerTenantStatistic.html url with the new one - https://docs.victoriametrics.com/pertenantstatistic/ 2024-04-18 03:36:23 +02:00
Aliaksandr Valialkin
4f1eb03441
all: replace old https://docs.victoriametrics.com/Quick-Start.html url with the new one - https://docs.victoriametrics.com/quick-start/ 2024-04-18 03:33:01 +02:00
Aliaksandr Valialkin
b1477aa2be
all: replace old https://docs.victoriametrics.com/scrape_config_examples.html url with the new one - https://docs.victoriametrics.com/scrape_config_examples/ 2024-04-18 03:29:54 +02:00
Aliaksandr Valialkin
87641fa7e7
all: replace old https://docs.victoriametrics.com/Troubleshooting.html url with the new one - https://docs.victoriametrics.com/troubleshooting/ 2024-04-18 03:27:18 +02:00