Commit Graph

4299 Commits

Author SHA1 Message Date
Github Actions
6f12ca71eb
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@47c26c4 (#6586)
(cherry picked from commit 7e781f0f78)
2024-07-05 09:49:32 +02:00
hagen1778
7d98c200eb
docs: update refereneces to victoriametrics-datasource plugin
The plugin was renamed in https://github.com/VictoriaMetrics/victoriametrics-datasource/pull/178

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit ee66fb4387)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-05 09:49:28 +02:00
Yury Molodov
7fc9912d15
vmui: add compact JSON display (#6582)
### Describe Your Changes
If a JSON element has only one field, it will be displayed on a single
line.
 #6559

| Old Display | New Display |
|-------------|-------------|
|
![image](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/8866517b-a49d-450f-904c-19117397a078)
|
![image](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/8e222b43-a4cb-4f32-9a79-6199778404d3)
|

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 959a4383c5)
2024-07-05 09:49:12 +02:00
Github Actions
9e324cc28a
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@b326811 (#6580)
(cherry picked from commit 7e169a767d)
2024-07-05 09:49:10 +02:00
Github Actions
8df41f37e6
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@4ed20bd (#6578)
(cherry picked from commit 11989db586)
2024-07-05 09:49:08 +02:00
Hui Wang
bbd49a1a61
vmalert: allow omitting -replay.timeTo in replay mode, default valu… (#6575)
…e is the current timestamp

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6492

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3169524fb7)
2024-07-05 09:49:06 +02:00
Roman Khavronenko
b13c363f12
app/vmalert: add examples for source override (#6561)
The change adds a new docs section with examples on how source can be
overridden. It should address questions like
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6536

While there, fix the example in `external.alert.source` cmd-line flag
and docker-compose examples.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c429bbf889)
2024-07-05 09:49:03 +02:00
Aliaksandr Valialkin
210e5b8957
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.1-victorialogs 2024-07-05 01:22:50 +02:00
Aliaksandr Valialkin
6397c38a0a
lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings
The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-05 01:22:49 +02:00
Github Actions
3d6e3ee98f
Automatic update operator docs from VictoriaMetrics/operator@0c311bb (#6574) 2024-07-04 11:46:16 +02:00
Aliaksandr Valialkin
172ae1adf7
Revert c6c5a5a186 and b2765c45d0
Reason for revert:

There are many statsd servers exist:

- https://github.com/statsd/statsd - classical statsd server
- https://docs.datadoghq.com/developers/dogstatsd/ - statsd server from DataDog built into DatDog Agent ( https://docs.datadoghq.com/agent/ )
- https://github.com/avito-tech/bioyino - high-performance statsd server
- https://github.com/atlassian/gostatsd - statsd server in Go
- https://github.com/prometheus/statsd_exporter - statsd server, which exposes the aggregated data as Prometheus metrics

These servers can be used for efficient aggregating of statsd data and sending it to VictoriaMetrics
according to https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd (
the https://github.com/prometheus/statsd_exporter can be scraped as usual Prometheus target
according to https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter ).

Adding support for statsd data ingestion protocol into VictoriaMetrics makes sense only if it provides
significant advantages over the existing statsd servers, while has no significant drawbacks comparing
to existing statsd servers.

The main advantage of statsd server built into VictoriaMetrics and vmagent - getting rid of additional statsd server.
The main drawback is non-trivial and inconvenient streaming aggregation configs, which must be used for the ingested statsd metrics (
see https://docs.victoriametrics.com/stream-aggregation/ ). These configs are incompatible with the configs for standalone statsd servers.
So you need to manually translate configs of the used statsd server to stream aggregation configs when migrating
from standalone statsd server to statsd server built into VictoriaMetrics (or vmagent).

Another important drawback is that it is very easy to shoot yourself in the foot when using built-in statsd server
with the -statsd.disableAggregationEnforcement command-line flag or with improperly configured streaming aggregation.
In this case the ingested statsd metrics will be stored to VictoriaMetrics as is without any aggregation.
This may result in high CPU usage during data ingestion, high disk space usage for storing all the unaggregated
statsd metrics and high CPU usage during querying, since all the unaggregated metrics must be read, unpacked and processed
during querying.

P.S. Built-in statsd server can be added to VictoriaMetrics and vmagent after figuring out more ergonomic
specialized configuration for aggregating of statsd metrics. The main requirements for this configuration:

- easy to write, read and update (ideally it should work out of the box for most cases without additional configuration)
- hard to misconfigure (e.g. hard to shoot yourself in the foot)

It would be great if this configuration will be compatible with the configuration of the most widely used statsd server.

In the mean time it is recommended continue using external statsd server.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6265
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5053
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5052
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4600
2024-07-03 23:57:49 +02:00
Github Actions
bc0303bca3
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@6182902 (#6572) 2024-07-03 15:22:50 +02:00
Github Actions
730088caaf
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@a04844a (#6571) 2024-07-03 15:22:50 +02:00
Github Actions
a72d0310c0
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@a53ccd7 (#6570) 2024-07-03 14:20:56 +02:00
Aliaksandr Valialkin
4268a310c1
app/vmagent/remotewrite/remotewrite.go: make remoteWriteCtx.TryPush code easier to follow
Move the code responsible for relabelCtx clearing into deferred function.
This allows making more clear the remoteWriteCtx.TryPush code.

This is a follow-up for 879771808b

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205

While at it, clarify the description of the bugfix at docs/CHANGELOG.md
2024-07-03 14:18:51 +02:00
Aliaksandr Valialkin
f406764ccc
app/vmagent/remotewrite/streamaggr.go: clarify the description for -remoteWrite.streamAggr.* command-line flags, so they are applied to the corresponding -remoteWrite.url 2024-07-03 14:18:51 +02:00
Artem Navoiev
f327eac81b
add file placeholder grafana logs datasource docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-03 14:18:50 +02:00
Artem Navoiev
7ddaf51414
remove unsued image
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-03 14:18:50 +02:00
Github Actions
3a11c84ee8
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@dae8560 (#6568)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-03 14:18:49 +02:00
Aliaksandr Valialkin
f9b64cdfe0
docs/MetricsQL.md: document which metric types are usually passed to which rollup functions
This should reduce incorrect usage of rollup functions like this one - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3974#issuecomment-2205574667
2024-07-03 11:36:58 +02:00
Aliaksandr Valialkin
547aa6108e
docs: update VictoriaMetrics release from v1.100.1 to v1.101.0 across all the docs
This is a follow-up for 5e8c087d42

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6194
2024-07-03 01:07:07 +02:00
Aliaksandr Valialkin
bb7406e9c0
app/vmselect/promql: follow-up for dd0d2c77c8 and 6149adbe10
Use metricsql.IsLikelyInvalid() function for determining whether the given query is likely invalid,
e.g. there is high change the query is incorrectly written, so it will return unexpected results.

The query is invalid most of the time if it passes something other than series selector into rollup function.
For example:

- rate(sum(foo))
- rate(foo + bar)
- rate(foo > bar)

Improtant note: the query is considered valid if it misses the lookbehind window in square brackes inside rollup function,
e.g. rate(foo), since this is very convenient MetricsQL extention to PromQL, and this query returns the expected results
most of the time.

Other unsafe query types can be added in the future into metricsql.IsLikelyInvalid().

TODO: probably, the -search.disableImplicitConversion command-line flag must be set by default in the future releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6180
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6450
2024-07-03 00:46:56 +02:00
Aliaksandr Valialkin
82748b2b9d
deployment/docker: update Go builder from Go1.22.4 to Go1.22.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved
2024-07-03 00:07:55 +02:00
Aliaksandr Valialkin
f8779d1ed2
lib/streamaggr: follow-up for the commit c0e4ccb7b5
- Clarify docs for `Ignore aggregation intervals on start` feature.

- Make more clear the code dealing with ignoreFirstIntervals at aggregator.runFlusher() functions.
  It is better from readability and maintainability PoV using distinct a.flush() calls
  for distinct cases instead of merging them into a single a.flush() call.

- Take into account the first incomplete interval when tracking the number of skipped aggregation intervals,
  since this behaviour is easier to understand by the end users.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6137
2024-07-02 21:34:48 +02:00
hagen1778
e423c4b72c
docs: mention graphite.sanitizeMetricName in cluster docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ffb49c677b)
2024-07-02 17:18:07 +02:00
Andrii Chubatiuk
252aa5a3ab
lib/protoparser/graphite: added -graphite.sanitizeMetricName flag (#6489)
### Describe Your Changes

Added flag to sanitize graphite metrics
fixes #6077

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 476faf5578)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-02 17:16:00 +02:00
Zakhar Bessarab
4ece1747d3
vmbackupmanager: fix state restore (#773)
* app/vmbackupmanager: fix state restore on startup

Fix improperly treating completed backups as "failed" in metrics.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: document the fix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f3831bdd13)
2024-07-02 14:37:15 +02:00
LHHDZ
c8431c8e4d
app/vmauth: reader pool to reduce gc & mem alloc (#6533)
follow up https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6446

issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 4d66e042e3)
2024-07-02 14:37:15 +02:00
Github Actions
3fda4cc4de
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@e9bd079 (#6564)
(cherry picked from commit dd97dd6373)
2024-07-02 14:37:15 +02:00
Aliaksandr Valialkin
fed596b202
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.0-victorialogs 2024-07-02 01:43:54 +02:00
Aliaksandr Valialkin
0912a652d5
app/vlinsert/insertutils: flush the ingested logs from in-memory buffer to storage every second
Previously the in-memory buffer could remain unflushed for long periods of time under low ingestion rate.
The ingested logs weren't visible for search during this time.
2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin
ab28a1f93e
app/vlinsert/syslog: add an ability to use log ingestion time as the _time field 2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin
f4dd9bd988
docs/VictoriaLogs/CHANGELOG.md: use new url https://docs.victoriametrics.com/victorialogs/querying/ instead of old one https://docs.victoriametrics.com/VictoriaLogs/querying/
(cherry picked from commit 387b3b7fb7)
2024-07-01 16:40:44 +02:00
Andrii Chubatiuk
65c742d976
deployment: remove snap packages support (#6543)
### Describe Your Changes

Removed snap packages support as it requires time for maintenance and
it's not popular at all

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0a42c8fd8b)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-01 16:40:39 +02:00
Hui Wang
87cb132f53
app/vmselect/netstorage: do not retry request when complexity limit i… (#6469)
…s already exceeded

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-01 16:38:15 +02:00
Andrii Chubatiuk
937ae2ca90
lib/streamaggr: added stale samples metric, added metrics labels (#6462)
### Describe Your Changes

- added stale metrics counters for input and output samples
- added labels for aggregator metrics =>
`name="{rwctx}:{aggrId}:{aggrSuffix}"`
   - rwctx - global or number starting from 1
   - aggrid - aggregator id starting from 1
   - aggrSuffix - <interval>_(by|without)_label1_label2_labeln
   e.g: `name="global:1:1m_without_instance_pod"`

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 861852f262)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-01 15:01:49 +02:00
hagen1778
69625aa8a1
docs: mark without optional in stream aggr docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f8eea0f2c9)
2024-07-01 14:56:46 +02:00
Aliaksandr Valialkin
5143e93534
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.1-victorialogs 2024-07-01 02:33:02 +02:00
Aliaksandr Valialkin
208a624d4d
lib/logstorage: properly search for the surrounding logs in stream_context pipe
The set of log fields in the found logs may differ from the set of log fields present in the log stream.
So compare only the log fields in the found logs when searching for the matching log entry in the log stream.

While at it, return _stream field in the delimiter log entry, since this field is used by VictoriaLogs Web UI
for grouping logs by log streams.
2024-07-01 02:33:00 +02:00
Aliaksandr Valialkin
516c7b2ca0
docs/VictoriaLogs: remove "preview" warning - VictoriaLogs is ready for prod 2024-07-01 01:52:40 +02:00
Aliaksandr Valialkin
0e6ee55a95
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.0-victorialogs 2024-07-01 01:49:55 +02:00
Aliaksandr Valialkin
76a58ae08d
lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:46:03 +02:00
Aliaksandr Valialkin
d0dca7b8c5
lib/logstorage: add delimiter between log chunks returned from | stream_context pipe 2024-07-01 01:46:02 +02:00
Artem Navoiev
2ac2d0919d
operator: sync docs with operator repo, add alias
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-06-29 17:19:49 +02:00
Aliaksandr Valialkin
76053a0ef0
docs/VictoriaLogs: typo fixes 2024-06-28 19:26:32 +02:00
Aliaksandr Valialkin
75c7b2c07a
docs/VictoriaLogs/CHANGELOG.md: cut v0.25.0-victorialogs 2024-06-28 19:18:21 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
c9fc8079c4
app/vlinsert/syslog: properly skip empty lines in Syslog protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548
2024-06-28 14:09:45 +02:00
Aliaksandr Valialkin
bb6424aeca
app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint
This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545
in order to return top N groups with the biggest number of hits.
2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
3eecc3de8c
docs/VictoriaLogs: typo fixes 2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
5a24ea6cc3
docs/VictoriaLogs: document how to export logs from VictoriaLogs
While at it, mention vmctl-like tool for migrating logs from other systems to VictoriaLogs at docs/VictoriaLogs/Roadmap.md

Thanks to the question from Xavier Pestel at https://www.linkedin.com/feed/update/urn:li:activity:7212093021301927937?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7212093021301927937%2C7212171550651731969%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287212171550651731969%2Curn%3Ali%3Aactivity%3A7212093021301927937%29
2024-06-28 02:06:24 +02:00
Aliaksandr Valialkin
05365999ea
docs/VictoriaLogs/querying/README.md: clarify live tailing docs 2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin
4391ef25e9
docs/VictoriaLogs/Roadmap.md: remove Web UI from roadmap, since it is already implemented
See https://docs.victoriametrics.com/victorialogs/querying/#web-ui

While at it, add DataDog protocol for logs into 'data ingestion protocols' roadmap.
2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin
00899506f7
docs/VictoriaLogs/querying: remove unused and outdated vmui.webp image
The image has been deleted from the docs/VictoriaLogs/querying/README.md file
in the commit 87f1c8bd6c
2024-06-27 15:16:43 +02:00
Aliaksandr Valialkin
13df75a28b
docs/VictoriaLogs/querying/README.md: add a link to live tailing docs from querying log docs 2024-06-27 15:12:59 +02:00
Aliaksandr Valialkin
b26acec9a8
app/vlselect: properly return live tailing results 2024-06-27 15:06:15 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
0352d98d74
docs/goals.md: clarify that the usability of the existing functionality must be improved the first 2024-06-27 09:55:40 +02:00
Josh Keife
bdebc128f7
Update vmalert.md (#6539)
Fix typo

### Describe Your Changes

Fix typo in the vmalert docs. In the docs it states rules at `VMAgent`'s
namespace when it should be `VMAlert`'s namespace.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 8d01dd0514)
2024-06-27 09:32:07 +02:00
Aliaksandr Valialkin
911891da95
docs/goals.md: clarify that the listed goals are related to development and support, not sales and marketing 2024-06-26 19:12:36 +02:00
Fred Navruzov
ed41f18c5a
docs/vmanomaly: quickstart page improvements (#6538)
### Describe Your Changes

Small improvements to a QuickStart guide of `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
Andrii Chubatiuk
580d02c3f8
added IMDSv2 for YC SD (#6524)
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
Andrii Chubatiuk
f79df2aa8b
app/vmauth: allow dropping host header (#6525)
### Describe Your Changes

Fixes #6453

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
rtm0
48a5c4cb01
Fix Date metricid cache consistency under concurrent use (#6534)
### Describe Your Changes

Fix Date metricid cache consistency under concurrent use.
When one goroutine calls Has() and does not find the cache entry in the
immutable map it will acquire a lock and check the mutable map. And it
is possible that before that lock is acquired, the entry is moved from
the mutable map to the immutable map by another goroutine causing a
cache miss.

The fix is to check the immutable map again once the lock is acquired. 

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-06-26 19:12:34 +02:00
Aliaksandr Valialkin
3a2c0b0a61
docs/goals.md: heading fixes 2024-06-26 16:41:21 +02:00
Aliaksandr Valialkin
d2be74b4a1
docs/goals.md: add VictoriaMetrics proverbs 2024-06-26 14:01:15 +02:00
Aliaksandr Valialkin
75d45068a5
docs/VictoriaLogs/README.md: document the relation between -retention.maxDiskSpaceUsageBytes and -retentionPeriod command-line flags 2024-06-26 12:03:29 +02:00
Aliaksandr Valialkin
db5a5a4f6e
docs/goals.md: add additional non-goals, which frequently become false goals according to the development history of VictoriaMetrics 2024-06-26 12:03:28 +02:00
Yury Molodov
6bde0196d8
vmui/logs: fix the update of the relative time range (#6517)
### Describe Your Changes

- Fixed the update of the relative time range when `Execute Query` is
clicked
- Optimized server requests: now, if an error occurs in the `/query`
request, the `/hits` request will not be executed.

#6345 (duplicates: #6440, #6312)

(cherry picked from commit 43342745ac)
2024-06-26 11:26:08 +02:00
James Rhoat
8249de986e
Updating documentation around OTEL (#6519)
Updating documentation around the opentelemetry endpoint for metrics and
the "How to use OpenTelemetry metrics with VictoriaMetrics" guide so
that it shows not only how to directly write but also how to write to
the otel collector and view metrics in vmui.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit 6652fb630f)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-26 11:26:05 +02:00
Yury Molodov
904ec020ed
vmui: fix input cursor position reset (#6530)
### Describe Your Changes

This PR addresses the issue where the cursor jumps to the end of the
input fields in the modal settings window after each keystroke.

### Before fix:

![ezgif-7-4c69805cea](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/2e99e833-09e3-4b44-89aa-fc1bd3c4346d)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit e9b71a2883)
2024-06-26 11:25:47 +02:00
hagen1778
6ad66b3317
docs: rm mentions of vmrecover as it still doesn't exist
While there, add a hint about truncated error messages.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 6775a00622)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-26 11:25:42 +02:00
Daria Karavaieva
affde1794f
docs/vmanomaly:quickstart example fix (#6535)
- Fixed config example in QuickStart vmanomaly docs for 1.13 version
compatibility

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 4b77778ff5)
2024-06-26 11:25:15 +02:00
Aliaksandr Valialkin
2dfbc40563
docs/goals.md: add VictoriaMetrics goals and non-goals page
Everybody who works on VictoriaMetrics must be aware of these goals and non-goals
2024-06-25 18:00:55 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
252ff71a49
docs/VictoriaLogs/data-ingestion/README.md: document that Rsyslog and Syslog-ng can be used for ingesting logs into VictoriaLogs 2024-06-25 15:15:53 +02:00
Aliaksandr Valialkin
0a88a55ae0
docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release 2024-06-25 14:59:31 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
hagen1778
26d58f8768
docs: add missing ticket links to the recent releases
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 14:54:24 +02:00
hagen1778
841305da8a
deployment: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0dcb3acbf7)
2024-06-25 13:58:59 +02:00
Nikolay
bf1464fc33
dashboards: add dashboard and alerts for vmauth (#6491)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 14b9ef1e4d)
2024-06-25 11:17:04 +02:00
hagen1778
8237b63fe3
docs: make docs-sync
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cc6a1da0d7)
2024-06-25 11:17:04 +02:00
Roman Khavronenko
e3da10a7c1
docs: add section for docs modification (#6528)
While there, remove the obsolete files from the previous version of
docs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 87edfa61c1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 11:17:03 +02:00
Aliaksandr Valialkin
a5445e09c2
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 5ecf439078.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin
813a7aec6c
docs: return back spellcheck fixes from the commit 92b22581e6, which were accidentally reverted in 45cf83cd8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470
2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow (#6470)"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin
7026498359
lib/httpserver: revert 9b7e532172
Reason for revert: this commit doesn't resolve real security issues,
while it complicates the resulting code in subtle ways (aka security circus).

Comparison of two strings (passwords, auth keys) takes a few nanoseconds.
This comparison is performed in non-trivial http handler, which takes thousands
of nanoseconds, and the request handler timing is non-deterministic because of Go runtime,
Go GC and other concurrently executed goroutines. The request handler timing is even
more non-deterministic when the application is executed in shared environments
such as Kubernetes, where many other applications may run on the same host and use
shared resources of this host (CPU, RAM bandwidth, network bandwidth).

Additionally, it is expected that the passwords and auth keys are passed via TLS-encrypted connections.
Establishing TLS connections takes additional non-trivial time (millions of nanoseconds),
which depends on many factors such as network latency, network congestion, etc.

This makes impossible to conduct timing attack on passwords and auth keys in VictoriaMetrics components.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6423/files
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392
2024-06-25 01:51:06 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
8dd69fd7ae
docs/CHANGELOG.md: cut v1.102.0-rc2 release 2024-06-24 23:22:45 +02:00
Nikolay
7ab046cf88
vendor update: updates metrics to v1.34.0 (#6523)
mention change for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1af13208c2)
2024-06-24 14:52:32 +02:00
Ivan Yatskevich
50343768fd
Publish Alertmanager configuration docs for Managed VictoriaMetrics (#6428)
### Describe Your Changes

With the recent release of Managed VictoriaMetrics users are able to
create and execute Alerting & Recording rules and send notifications via
hosted Alertmanager.

So, we're publishing Alertmanager configuration docs for Managed
VictoriaMetrics.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1106d91741)
2024-06-24 12:50:15 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow (#6470)
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Andrii Chubatiuk
50783fca4d
app/vmagent: add max_scrape_size to scrape config (#6434)
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1e83598be3)
2024-06-20 14:00:22 +02:00
Hui Wang
a393b993d6
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512

(cherry picked from commit bc37b279aa)
2024-06-20 14:00:19 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
Zakhar Bessarab
886f545f81
lib/fs/fscore: do not trim content from path (#6503)
### Describe Your Changes

Trimming content which is loaded from an external pass leads to obscure
issues in case user-defined input contained trimmed chars. For example.
user-defined password "foo\n" will become "foo" while user will expect
it to contain a new line.

---
For example, a user defines a password which ends with `\n`. This often
happens when user Kubernetes secrets and manually encodes value as
base64-encoded string.

In this case vmauth configuration might look like:
```
users:
  - url_prefix:
      - http://vminsert:8480/insert/0/prometheus/api/v1/write
    name: foo
    username: foo
    password: "foobar\n"
```

vmagent configuration for this setup will use the following flags:
```
-remoteWrite.url=http://vmauth:8427/
-remoteWrite.basicAuth.passwordFile=/tmp/vmagent-password
-remoteWrite.basicAuth.username="foo"
```
Where `/tmp/vmagent-password` is a file with `foobar\n` password.

Before this change such configuration will result in `401 Unauthorized`
response received by vmagent since after file content will become
`foobar`.

---
An example with Kubernetes operator which uses a secret to reference the
same password in multiple configurations.

<details>
  <summary>See full manifests</summary>

`Secret`:
```
apiVersion: v1
data:
  name: Zm9v # foo
  password: Zm9vYmFy # foobar\n
  username: Zm9v= # foo
kind: Secret
metadata:
  name: vmuser
```

`VMUser`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
  name: vmagents
spec:
  generatePassword: false
  name: vmagents
  targetRefs:
  - crd:
      kind: VMAgent
      name: some-other-agent
      namespace: example
  username: foo
  # note - the secret above is referenced to provide password
  passwordRef:
    name: vmagent
    key: password
```

`VMAgent`:
```
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: example
spec:
  selectAllByDefault: true
  scrapeInterval: 5s
  replicaCount: 1
  remoteWrite:
    - url: "http://vmauth-vmauth-example:8427/api/v1/write"
      # note - the secret above is referenced as well
      basicAuth:
        username:
          name: vmagent
          key: username
        password:
          name: vmagent
          key: password
```

</details>

Since both config target exactly the same `Secret` object it is expected
to work, but apparently the result will be `401 Unauthrized` error.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 201fd6de1e)
2024-06-19 10:37:12 +02:00
Nihal
8fd46caa22
victoria-metrics: constant-time comparison of credentials like authkeys and basic auth credentials (#6423)
Changes for constant-time comparison of credentials like authkeys and
basic auth credentials.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392

---------

Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
(cherry picked from commit 9b7e532172)
2024-06-19 10:37:09 +02:00
Hui Wang
36f6a08f07
doc: add missing cmd flag description (#6504)
(cherry picked from commit 2259e3cac7)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-18 15:29:11 +02:00
Yury Molodov
88650abf97
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.

(cherry picked from commit 32fbffedd9)
2024-06-18 15:28:56 +02:00