Commit Graph

8160 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
55bd43f28e
docs: follow-up for ac9c2a796f
Remove description for -search.maxExportDuration and -search.maxStatusRequestDuration command-line flags
from the 'Resource usage limits' chapter, since these flags are rarely used for limiting resource usage
and they are already documented in the 'List of command-line flags' chapter.
2024-04-02 23:57:37 +03:00
Aliaksandr Valialkin
e4eccd7074
app/vmselect/graphite: follow-up for 23ab865035
- Fix docs for new functions at app/vmselect/graphite/functions.json
- Properly drain series lists on errors in aggregateSeriesListsGeneric() and aggregateSeriesList()
- Add links to docs for the added functions at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5809
2024-04-02 23:39:00 +03:00
Aliaksandr Valialkin
918cccaddf
all: fix golangci-lint(revive) warnings after 0c0ed61ce7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001
2024-04-02 23:16:29 +03:00
Aliaksandr Valialkin
c3a72b6cdb
lib/storage: consistently use stopCh instead of stop 2024-04-02 21:24:57 +03:00
Aliaksandr Valialkin
be36ceb1cf
app/vmauth: add ability to authorize via any opaque HTTP request header value
This can be done via `auth_token` option at -auth.config - see https://docs.victoriametrics.com/vmauth/#auth-config
2024-04-02 21:16:11 +03:00
Aliaksandr Valialkin
21bfb66650
app/vmauth: add ability to read auth tokens from multiple http request headers
This is needed for VictoriaMetrics Cloud, where the same token could be passed either
via Authorization or via X-Amz-Firehose-Access-Key header - see 4487dac30b (r140500722)

This is a follow-up for 4487dac30b

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-02 19:29:00 +03:00
Artem Navoiev
9bd3cadce6
app/{vmagent/insert} fix typo in Firehose
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-04-02 17:41:21 +02:00
Aliaksandr Valialkin
4487dac30b
app/vmauth: follow-up for bc90f4aae6
- Allow specifying only a single HTTP header for reading auth tokens via -httpAuthHeader command-line flag.
  This is better from security PoV, since this prevents from accidental reading of auth token from undesired
  HTTP header. By default the -httpAuthHeader equals to Authorization. When it is overridden, then
  auth token isn't read from Authorization header - it is read only from the specified header.

- Document the -httpAuthHeader command-line flag at https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-02 18:35:21 +03:00
Aliaksandr Valialkin
904e95fc69
app/vmagent: simplify code after 509df44d03
- Simplify the code in order to improve its maintenance
- Properly pass tenant ID when processing multi-tenant opentelemetry request at vmagent

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6016
2024-04-02 17:58:13 +03:00
Artem Navoiev
76b1fc6ac1
add more delays to verify that this is not a reason for flaky tests
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-04-02 16:22:36 +02:00
Fred Navruzov
daa1326b98
docs/vmanomaly: typos fix (#6047) 2024-04-01 13:23:44 -07:00
Fred Navruzov
c300ce659f
docs/vmanomaly: v1.12 updates & fixes (#6046)
* docs/vmanomaly: v1.12.0 & link updates

* add autotuned description to model section

* - update refs of vmanomaly on enterprise and vmalert pages
- add diagrams for model types
- update self-monitoring section

* - fix typos
- remove .index.html from links
2024-04-01 16:41:55 +03:00
Aliaksandr Valialkin
c79bf3925c
Revert "app/vmselect: make vmselect resilient to absence of cache folder (#5987)"
This reverts commit cb23685681.

Reason for revert: the "fix" may hide programming bugs related to incorrect creation of folders
before their use. This may complicate detecting and fixing such bugs in the future.

There are the following fixes for the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985 :
- To configure the OS to do not drop data from the system-wide temporary directory (aka /tmp).
- To run VictoriaMetrics with -cacheDataPath command-line flag, which points to the directory,
  which cannot be removed automatically by the OS.

The case when the user accidentally deletes the directory with some files created by VictoriaMetrics
shouldn't be considered as expected, so VictoriaMetrics shouldn't try resolving this case automatically.
It is much better from operation and debuggability PoV is to crash with the clear `directory doesn't exist` error
in this case.
2024-03-30 07:29:24 +02:00
Aliaksandr Valialkin
49a6dca2d5
docs/CHANGELOG.md: mention that the bug with improper use of -search.maxExportDuration instead of -search.maxLabelsAPIDuration has been introduced in v1.99.0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5992
This is a follow-up for bc79f7196d
2024-03-30 07:05:09 +02:00
Aliaksandr Valialkin
8f59ca423b
docs/VictoriaLogs/CHANGELOG.md: improve the description of the bugfix from 43b5d8bc7a, so it can be googled by users 2024-03-30 06:54:48 +02:00
Aliaksandr Valialkin
830b871baf
app/vmagent: properly shutdown when -maxIngestionRate limit is reached
The remotewrite.Stop() expects that there are no pending calls to TryPush().
This means that the ingestionRateLimiter.Register() must be unblocked inside TryPush() when calling remotewrite.Stop().
Provide remotewrite.StopIngestionRateLimiter() function for unblocking the rate limiter before calling the remotewrite.Stop().

While at it, move the rate limiter into lib/ratelimiter package, since it has two users.
Also move the description of the feature to the correct place at docs/CHANGELOG.md.
Also cross-reference -remoteWrite.rateLimit and -maxIngestionRate command-line flags.

This is a follow-up for 02bccd1eb9
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900
2024-03-30 06:43:48 +02:00
Aliaksandr Valialkin
f5848a5c8b
docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md: user proper image paths according to docs/assets/README.md 2024-03-30 05:09:41 +02:00
Aliaksandr Valialkin
f17248eb3f
docs/managed-victoriametrics: use proper names for the linked images according to docs/assets/README.md
This is a follow-up for db3709c87d

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5989
2024-03-30 05:00:19 +02:00
Aliaksandr Valialkin
4cb70ee9a3
vendor: update github.com/VictoriaMetrics/metrics and github.com/VictoriaMetrics/metricsql to newer versions
This is needed for updating broken links to MetricsQL docs:

https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL -> https://docs.victoriametrics.com/metricsql/

This is a follow-up for 7e3511ffbd
2024-03-30 04:44:19 +02:00
Aliaksandr Valialkin
4d71a33cb5
docs/CHANGELOG.md: remove the update notes regarding converting custom HTTP header keys to canonical form
Custom HTTP headers are set via net/http.Header.Set or net/http.Header.Add functions.
These functions always convert header keys to canonical form. So the change at b577413d3b
isn't visible to users of VictoriaMetrics components.

There is no need in documenting this change at docs/CHANGELOG.md, since it doesn't give any useful information to users.

This is a follow-up for e6dd52b04c
2024-03-30 04:26:37 +02:00
Zakhar Bessarab
af3922b1df
lib/storage: add ability to use downsampling for the given series filter (#733)
* lib/storage: add ability to use downsampling for the given series filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add information about downsampling filters

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: fix MetricsQL filter

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: treat missing downsampling filter as a bug

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/part_header: verify correctness of downsampling filters when opening partition

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: save only appliable rules in part metadata

Filter and save only rules which are appliable to partition based on MinTimestamp of stored data.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage/downsampling: update log messages for final dedup

Properly specify a reason of re-running deduplication for partition.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/storage: consistently use MaxTimestamp to determine deduplication/downsampling rules

Using MinTimestamp leads to applying downsampling to parts which are only partially covered by downsampling rule.
For example, partition covers range [1000-2000]. At t=2100 and rule offset 500 data with t=2100-500 => 1600 must be downsampled. The range check against MinTimestamp evaluates to true even though partition contains range which must not be downsampled - [1600:2000].

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Follow-up

- Apply the first matching downsampling period if multiple filters match the given time series.
  This allows fine-tuning the downsampling config for the specific needs.
- Take into account downsampling filters during search queries.
- Reduce the difference between community and enterprise branches. This should simplify further maintenance of these branches.
- Properly parse series filters with colons inside them.
- Document the feature at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-03-30 04:12:23 +02:00
Aliaksandr Valialkin
131f357098
lib/storage/table.go: reduce the difference with enterprise branch 2024-03-30 03:22:51 +02:00
Aliaksandr Valialkin
4001ca36b8
lib/storage/partition.go: reduce code difference a bit with enterprise branch 2024-03-30 01:39:27 +02:00
Nikolay
a05303eaa0
lib/storage: adds metrics for downsampling (#382)
* lib/storage: adds metrics for downsampling
vm_downsampling_partitions_scheduled - shows the number of parts, that must be downsampled
vm_downsampling_partitions_scheduled_size_bytes - shows total size in bytes for parts, the must be donwsampled

These two metrics answer the questions - is downsampling running? how many parts scheduled for downsampling and how many of them currently downsampled? Storage space that it occupies.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-03-30 01:11:49 +02:00
hagen1778
e79b05b4ab
docs: update vmalert troubleshooting docs
* rm recommendation to keep look-behind window empty, as it is not correct
* mention the change of default value for `-search.latencyOffset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-29 18:00:37 +01:00
hagen1778
2e843a8ed9
docs: follow-up after 623d257faf
623d257faf
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-29 14:29:02 +01:00
Jiekun
623d257faf
app/vmalert: respect batch size limit for remote write on shutdown (#6039)
During shutdown period of vmalert, remotewrite client retrieve all pending time series from buffer queue, compose them into 1 batch and execute remote write.

This final batch may exceed the limit of -remoteWrite.maxBatchSize, and be rejected by the receiver (gateway, vmcluster or others). 

This changes ensures that even during shutdown vmalert won't exceed the max batch size limit for remote write
destination.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025
2024-03-29 14:27:50 +01:00
hagen1778
b6bd9a97a3
app/vmagent: follow-up 166b97b8d0
* add tests for sharding function
* update flags description
* add changelog note

166b97b8d0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-29 14:08:08 +01:00
Andrii Chubatiuk
47892b4a4c
opentelemetry: added cmd flag to sanitize metric names (#6035) 2024-03-29 13:51:24 +01:00
Eugene Ma
166b97b8d0
vmagent: support sharding by excluded labels (#5938)
To horizontally scale streaming aggregation, you might want to deploy a separate hashing tier 
of vmagents that route to a separate aggregation tier. The hashing tier should shard by all labels 
except the instance-level labels, to ensure the input metrics are routed correctly to the aggregator 
instance responsible for those labels.
For this to achieve we introduce `remoteWrite.shardByURL.inverseLabels` flag to inverse logic of `remoteWrite.shardByURL.labels`

---------

Co-authored-by: Eugene Ma <eugene.ma@airbnb.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-03-29 13:26:02 +01:00
Dmytro Kozlov
ac9c2a796f
docs: describe timeout query argument (#6020)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-03-28 16:17:33 +01:00
Hui Wang
47e7ad2e01
docs: fix golangci-lint check (#6036)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-03-28 08:58:27 +01:00
Hui Wang
d7224b2d1c
vmalert: fix sending alert messages (#6028)
* vmalert: fix sending alert messages
1. fix `endsAt` field in messages that send to alertmanager, previously rule with small interval could never be triggered;
2. fix behavior of `-rule.resendDelay`, before it could prevent sending firing message when rule state is volatile.

* docs: update changelog notes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-03-28 08:55:10 +01:00
Aliaksandr Valialkin
77eca6bb37
docs/MetricsQL.md: typo fix: outlier_iqr_over_time(memory_usage_bytes[1h]) triggers when memory_usage_bytes goes outside the usual value range for the last hour, not the last 24 hours
This is a follow-up for ea81f6fc36
2024-03-27 20:59:43 +02:00
hagen1778
f937439657
docs: fix new line for update notes in CHANGELOG
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-27 16:24:16 +01:00
hagen1778
d72b565c03
docs: mention new guide How to use OpenTelemetry metrics with VictoriaMetrics in docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-27 16:23:28 +01:00
Nikolay
f8f4025dca
docs/opentelemetry: adds opentemetry get started guide (#5861)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
2024-03-27 16:04:43 +01:00
Aliaksandr Valialkin
4a359d5f67
lib/storage: follow-up for 76f00cea6b
Store the deadline when the metricID entries must be deleted from indexdb
if metricID->metricName entry isn't found after the deadline. This should
make the code more clear comparing the the previous version, where the timestamp
of the first metricID->metricName lookup miss was stored in missingMetricIDs.

Remove the misleading comment about the importance of the order for creating entries
in the inverted index when registering new time series. The order doesn't matter,
since any subset of the created entries can become visible for search
before any other subset after registering in indexdb.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959
2024-03-27 11:41:28 +02:00
Github Actions
20d0183195
Automatic update operator docs from VictoriaMetrics/operator@c336013 (#6023) 2024-03-26 17:55:02 +01:00
Github Actions
1263cab870
Automatic update operator docs from VictoriaMetrics/operator@ac29c88 (#6022) 2024-03-26 23:35:41 +08:00
hagen1778
5ffece51bf
docs: follow-up after 23ab865035
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-26 15:17:16 +01:00
rbizos
23ab865035
adding AggregateSeriesLists graphite function (#5809)
* adding aggregate series list graphite function

adding also aliases for sum diff and multiply

* Adding tests for aggregateSeriesLists and aliases
2024-03-26 15:13:34 +01:00
Zakhar Bessarab
51f5ac1929
lib/storage/table: wait for merges to be completed when closing a table (#5965)
* lib/storage/table: properly wait for force merges to be completed during shutdown

Properly keep track of running background merges and wait for merges completion when closing the table.
Previously, force merge was not in sync with overall storage shutdown which could lead to holding ptw ref.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add changelog entry

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-03-26 13:49:09 +01:00
Andrii Chubatiuk
bc90f4aae6
vmauth: support other auth header names besides Authorization (#6009) 2024-03-26 13:21:07 +01:00
Andrii Chubatiuk
509df44d03
app/{vmagent,vminsert}: fixed firehose response (#6016) 2024-03-26 13:20:41 +01:00
Roman Khavronenko
cb23685681
app/vmselect: make vmselect resilient to absence of cache folder (#5987)
vmselect uses a cache folder in file system for two purposes:
1. Storing rollup cache results on shutdown;
2. Storing temporary search results from vmstorage during query executions.

It could happen that cache folder is deleted accidentally by user, or by OS
during cleanup routines. This would cause vmselect to:
1. panic on /metrics call, because `MustGetFreeSpace` will fail;
2. return query error user, as it won't be able to store temporary search results.

The changes in this commit are the following:
1. Make `MustGetFreeSpace` to try re-creating the cache folder if it is missing;
2. Make vmselect to try re-creating the cache folder if it can't persist tmp search
results.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-03-26 12:59:50 +01:00
hagen1778
bc79f7196d
docs: follow-up for 70eaa06f08
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-03-25 15:30:56 +01:00
kbweave
70eaa06f08
app/vmselect: use GetDeadlineForLabelsAPI for LabelAPI requests (#5992) 2024-03-25 15:07:34 +01:00
Artem Navoiev
b08cbd0400
add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-03-25 10:09:15 +01:00
Artem Navoiev
b569fa0b2c
fix typo in kyiv city name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-03-23 21:56:31 +01:00