Commit Graph

2470 Commits

Author SHA1 Message Date
Artem Navoiev
34c705988e fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-27 13:17:22 +02:00
Artem Navoiev
7d9c4bebc0
update links to grafana dashboards (#3534)
docs: update links to grafana dashboards

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-25 17:36:20 +01:00
Aliaksandr Valialkin
0dca224ec3
app/vmui: small improvements for header panel
- Rename `Custom panel` tab to more clear `Query` tab
- Rename `Cardinality` tab to `Explore cardinality`, so it becomes consistent with `Explore metrics` tab
- Move `Dashboards` tab to the end, since it isn't used too much
2022-12-23 22:24:31 -08:00
Aliaksandr Valialkin
4deea604bf
app/vmui: follow-up after f6d31f5216
- Document the feature at docs/CHANGELOG.md.
- Document the metrics explorer at https://docs.victoriametrics.com/#metrics-explorer .
- Properly set `start` and `end` args for the selected time range
  when performing the request, which returns metric names.
- Improve queries, so they return lower number of lines and labels.
  This should improve metrics' exploration.
- Properly encode label filters and query args before passing them to VictoriaMetrics.
- Various cosmetic fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-22 17:17:01 -08:00
Yury Molodov
f6d31f5216
vmui: add explore tab for exploration of metrics, which belong to a particular job/instance (#3470)
* feat: add "Explore" page

* feat: add graphs for explore page

* vmui: add explore tab for exploration of metrics, which belong to a particular job/instance

* refactor: rename variables

* refactor: extract graph to ExploreMetricItemGraph.tsx

* feat: add searchable for Select.tsx

* feat: improve metrics explorer

* feat: set document title by page

* feat: add page to view icons

* fix: improve styles

* fix: add encodeURIComponent to query
2022-12-22 15:24:40 -08:00
Aliaksandr Valialkin
f6ac045933
docs/CHANGELOG.md: document 1df87807fd
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3513
2022-12-22 15:22:09 -08:00
Aliaksandr Valialkin
31886aef3d
lib/promrelabel: add support for keepequal and dropequal relabeling actions
These actions are supported by Prometheus starting from v2.41.0

See https://github.com/prometheus/prometheus/pull/11564 ,
https://github.com/prometheus/prometheus/issues/11556
and https://github.com/prometheus/prometheus/issues/3756

Side note:

It's a pity that Prometheus developers decided inventing `keepequal` and `dropequal`
relabeling actions instead of adding support for `keep_if_equal` and `drop_if_equal` relabeling
actions supported by VictoriaMetrics since June 2020 - see 2a39ba639d .
2022-12-21 20:04:55 -08:00
Yury Molodov
0f8ffc7df9
vmui: fix change timezone (#3519)
vmui: fix time picker with changed time zone
2022-12-21 17:22:37 +01:00
Aliaksandr Valialkin
f443fad56d
docs/CHANGELOG.md: cut v1.85.3 2022-12-20 14:51:23 -08:00
Roman Khavronenko
e4e9dfb785
vmagent: respect -usePromCompatibleNaming if no relabeling is set (#3511)
* vmagent: respect `-usePromCompatibleNaming` if no relabeling is set

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: upd test

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-20 14:32:45 -08:00
Yury Molodov
731d189fa9
fix: change the logic for hide query (#3514) 2022-12-20 14:29:46 -08:00
Zakhar Bessarab
4be4645142
app/vmbackupmanager: add metrics for better observability (#488)
* app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response

* app/vmbackupmanager: drop old metrics before creating new ones

* app/vmbackupmanager: use `_total` postfix for counter metrics

* app/vmbackupmanager: remove `_total` postfix for gauge-like metrics

* app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention

* app/vmbackupmanager: address review feedback

* app/vmbackupmanager: fix metric name

* app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage

* app/vmbackupmanager: improve performance for backup size calculation

* app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic

* {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package

* app/vmbackupmanager: fix creating new metrics instead of reusing existing ones

* lit/formatutil: add comment to make linter happy

* app/vmbackupmanager: address review feedback
2022-12-20 14:18:06 -08:00
Aliaksandr Valialkin
4e55b67a44
lib/storage: clear the err if it is set to io.EOF when searching for the TSID by metricID
This is expected error after when recently added indexdb data isn't available for search yet
or wasn't flushed to disk after unclean shutdown of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515
2022-12-20 14:05:29 -08:00
Aliaksandr Valialkin
a48510573e
Revert "docs/Release-Guide.md: add LATEST_TAG=stable env var for make publish-release in order to create stable tag for the published components at DockerHub"
This reverts commit 8afa7ef837.
2022-12-20 12:24:27 -08:00
Aliaksandr Valialkin
8afa7ef837
docs/Release-Guide.md: add LATEST_TAG=stable env var for make publish-release in order to create stable tag for the published components at DockerHub
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2911
2022-12-20 12:22:42 -08:00
Aliaksandr Valialkin
1d7b5cb83c
docs/CHANGELOG.md: document the change at 547f07463b29c09c62c9af35eac9cee6764b3286
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612
2022-12-20 10:22:07 -08:00
Roman Khavronenko
8122191368
docs: fix link typo in operator docs (#3508) 2022-12-20 14:52:47 +01:00
Aliaksandr Valialkin
6bf46c7bf5
docs/CHANGELOG.md: formatting fix 2022-12-20 01:06:34 -08:00
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
eabb8762ee
docs/Articles.md: add a link to https://www.youtube.com/watch?v=Mesc6JBFNhQ 2022-12-19 21:40:51 -08:00
Aliaksandr Valialkin
680925f872
docs/CHANGELOG.md: add a warning for releases between v1.83.0 and v1.85.1 that it is recommended upgrading to v1.85.2 because of the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
2022-12-19 13:36:49 -08:00
Aliaksandr Valialkin
d2ad184377
docs/CHANGELOG.md: consistently use YYYY-MM-DD format for release dates
The previously used DD-MM-YYYY format could be confused with the MM-DD-YYYY format.
The YYYY-MM-DD format reduces this confusion.
2022-12-19 13:33:05 -08:00
Aliaksandr Valialkin
6530344a8f
docs/CHANGELOG.md: cut v1.85.2 2022-12-19 13:09:29 -08:00
Aliaksandr Valialkin
0bf3ae9559
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname>
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
2022-12-19 13:08:11 -08:00
Aliaksandr Valialkin
6c98b56935
lib/storage: search for TSIDs for the given metricIDs in the previous indexdb if they aren't found in the current indexdb
The issue triggers after the indexdb rotation for time series, which stop receiving new samples.
This results in missing data for such time series in query responses.

This commit should address the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502

The issue has been introduced in 2dd93449d8
2022-12-19 12:03:09 -08:00
Aliaksandr Valialkin
4cb83f0f4a
lib/logger: follow-up for 72f8fce107
- Document the change at docs/CHANELOG.md
- Log fatal errors if the -loggerJSONFields contains unexpected values
- Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348
2022-12-16 17:42:07 -08:00
Aliaksandr Valialkin
56a9ea3753
docs/vmalert.md: mention latency_offset query arg, which has been added in 86dae56bd0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:20:37 -08:00
Aliaksandr Valialkin
65f8fc527f
lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page 2022-12-16 17:02:41 -08:00
Roman Khavronenko
86dae56bd0
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 16:54:57 -08:00
Aliaksandr Valialkin
17a244571b
docs/keyConcepts.md: update the list of supported data ingestion protocols
- Add DataDog protocol
- Remove native protocol, since it isn't intended for general-purpose usage by external clients
2022-12-15 12:01:59 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
5d30080555
lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes 2022-12-14 17:52:32 -08:00
Aliaksandr Valialkin
38341802c2
app/vmselect: add /expand-with-exprs page 2022-12-14 16:18:55 -08:00
Aliaksandr Valialkin
231569f89b
docs/vmagent.md: small formatting fix 2022-12-14 14:25:35 -08:00
Aliaksandr Valialkin
0c54ff20eb
docs/CHANGELOG.md: fix the link to the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 2022-12-14 14:17:17 -08:00
Aliaksandr Valialkin
76445bfd98
docs/CHANGELOG.md: add release date for v1.85.1 2022-12-14 14:12:07 -08:00
Aliaksandr Valialkin
ea44c39377
docs/CHANGELOG.md: cut v1.85.1 2022-12-14 11:57:49 -08:00
Aliaksandr Valialkin
e0009ec466
docs/Cluster-VictoriaMetrics.md: mention the vm_storage_is_read_only metric, which can help debugging readonly mode at vmstorage 2022-12-14 09:31:28 -08:00
Aliaksandr Valialkin
7a61bafe59
docs/vmagent.md: clarify that relabeling is actually a debugging at relabel debug section 2022-12-13 15:46:21 -08:00
Aliaksandr Valialkin
b89d862aa5
docs/CHANGELOG.md: document the bugfix at a50120a212 2022-12-13 09:36:24 -08:00
Dima Lazerka
bde93844de
Add Anomaly Detection to enterprise features list (#3476)
* Add Anomaly Detection to enterprise features list

* Update docs/enterprise.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-12 17:42:44 -08:00
Aliaksandr Valialkin
d1af6046c7
lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead
Blocked small merges may result into big number of small parts, which, in turn,
may result in increased CPU and memory usage during queries, since queries need to inspect
all the existing small parts.

The issue has been introduced in 8189770c50

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-12 17:00:50 -08:00
Aliaksandr Valialkin
3b18931050
lib/bytesutil: cache results for all the input strings, which were passed during the last 5 minutes from FastStringMatcher.Match(), FastStringTransformer.Transform() and InternString()
Previously only up to 100K results were cached.
This could result in sub-optimal performance when more than 100K unique strings were actually used.
For example, when the relabeling rule was applied to a million of unique Graphite metric names
like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466

This commit should reduce the long-term CPU usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466
after all the unique Graphite metrics are registered in the FastStringMatcher.Transform() cache.

It is expected that the number of unique strings, which are passed to FastStringMatcher.Match(),
FastStringTransformer.Transform() and to InternString() during the last 5 minutes,
is limited, so the function results fit memory. Otherwise OOM crash can occur.
This should be the case for typical production workloads.
2022-12-12 14:41:13 -08:00
Zakhar Bessarab
ebcb4ab617
{app/{vmbackup/vmrestore}: update path example to use Azure terminology for consistency (#3475) 2022-12-12 22:17:22 +03:00
Roman Khavronenko
8286f9608f
vmalert: support $for or .For template variables (#3474)
support `$for` or `.For` template variables  in alert's annotations.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-12 22:16:10 +03:00
Aliaksandr Valialkin
7ae744fce6
lib/protoparser/datadog: do not re-use previously parsed field values if they are missing in the currently parsed message
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-11 13:09:25 -08:00
Aliaksandr Valialkin
9f642d10ff
docs/CHANGELOG.md: cut v1.85.0 2022-12-11 02:01:11 -08:00
Aliaksandr Valialkin
38f8e8adc3
docs/CHANGELOG.md: document changes at v1.79.6 2022-12-11 01:51:36 -08:00
Aliaksandr Valialkin
d141cb28a6
docs/CHANGELOG.md: document 461158a437
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427
2022-12-10 23:42:16 -08:00
Aliaksandr Valialkin
88597f187b
app/{vmagent,vminsert}/datadog: make the host label optional in DataDog data ingestion protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-10 23:32:31 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
b7aec1be4d
docs/CHANGELOG.md: add a link to the issue related to reduced CPU and memory usage at vmalert
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3464

The related commit - b97bd01605
2022-12-10 22:21:19 -08:00
Aliaksandr Valialkin
b01607e3fb
docs: clarify that single-node VictoriaMetrics also provides functionality for relabel debugging
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2022-12-10 20:49:52 -08:00
Aliaksandr Valialkin
3f4cb9a142
docs: sync with cluster branch after 97b41e727c 2022-12-10 02:32:24 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
c5dd973f9c
docs/url-examples.md: add missing whitespace for proper heading for the example on how to send data via OpenTSDB protocol 2022-12-09 17:33:44 -08:00
Aliaksandr Valialkin
765ee5f7ba
docs/CHANGELOG.md: document b97bd01605 2022-12-09 11:49:29 -08:00
Aliaksandr Valialkin
2406c0dcfd
docs/CHANGELOG.md: document the bugfix at 05b42601c3
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
2022-12-08 18:35:28 -08:00
Aliaksandr Valialkin
5b9e6b9d24
lib/storage: follow-up after 7c0ae3a86a
- Update docs at https://docs.victoriametrics.com/#deduplication
- Optimize the deduplication loop a bit

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
2022-12-08 18:16:57 -08:00
Roman Khavronenko
7c0ae3a86a
lib/storage: keep sample with the biggest value on timestamp conflict (#3421)
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:06:11 -08:00
Aliaksandr Valialkin
eeacbaf0b6
all: update Go builder from v1.19.3 to v1.19.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved
2022-12-08 16:41:24 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
bce1c5d572
docs/Cluster-VictoriaMetrics.md: typo fix 2022-12-07 12:26:34 -08:00
Aliaksandr Valialkin
189217a069
docs/CHANGELOG.md: document the addition of file-based discovery of vmstorage nodes 2022-12-07 12:04:57 -08:00
Aliaksandr Valialkin
59430e4274
docs/Cluster-VictoriaMetrics.md: make docs-sync after 5de8330ce00adfc5ac794070d30a2617ddc14bf2 2022-12-07 12:02:38 -08:00
Aliaksandr Valialkin
49ebc48809
lib/querytracer: put the version of VictoriaMetrics in the first message of query trace
This should simplify further debugging, since the first thing to start the debugging by query trace
is to know the version of VictoriaMetrics, which produced this trace.
2022-12-07 09:46:39 -08:00
Aliaksandr Valialkin
3f82e3fa36
docs: follow-up after e1bf2a85d0559d112908ce81597f3261d3a085c0
- Document the change at docs/CHANGELOG.md
- Run `make docs-sync` for copying app/vmgateway/README.md to docs/vmgateway.md
  in order to propagate docs' changes to https://docs.victoriametrics.com/vmgateway.html
2022-12-06 21:05:22 -08:00
Aliaksandr Valialkin
35a3170d97
docs: document the addition of -storageNode.discoveryInterval command-line flag in VictoriaMetrics cluster enterprise
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3417
2022-12-06 19:57:06 -08:00
Aliaksandr Valialkin
5cc19b1f7e
docs/Articles.md: change the link to How we tried using VictoriaMetrics and Thanos at the same time from Russian to English article 2022-12-06 16:30:10 -08:00
Roman Khavronenko
3dec847c93
vmalert: correctly return error for RW failures (#3452)
* vmalert: correctly return error for RW failures

By mistake, in 0989649ad0 the error
for remote write failures weren't return to user.
This change fixes it.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-06 15:36:46 +01:00
Aliaksandr Valialkin
9a7c36e645
docs/Articles.md: add a link to the talk "How do We Keep Metrics for a Long Time in VictoriaMetrics" 2022-12-06 00:55:38 -08:00
Aliaksandr Valialkin
50ea632bfe
docs/Articles.md: link to a video for the talk 'VictoriaMetrics: scaling to 100 million metrics per second' 2022-12-06 00:53:05 -08:00
Aliaksandr Valialkin
a40c50f4fe
docs/CHANGELOG.md: document 1e0666abb4 2022-12-05 23:10:17 -08:00
Aliaksandr Valialkin
caa1c43166
docs: follow-up for 7645d9ae00
- Document the change at docs/CHANGELOG.md
- Document the feature at https://docs.victoriametrics.com/#vmui

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3446
2022-12-05 22:50:41 -08:00
Yury Molodov
01a9b36a95
vmui: timezone select (#3414)
* feat: add timezone selection

* vmui: provide feature timezone select

* fix: correct timezone with relative time

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:44:31 -08:00
Aliaksandr Valialkin
718d1d90b6
docs/CHANGELOG.md: document fd43b5bad0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445
2022-12-05 22:01:42 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Aliaksandr Valialkin
eed32b368c
docs/vmctl.md: make docs-sync after 86c31f2955
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2930
2022-12-05 17:24:10 -08:00
Zakhar Bessarab
86c31f2955
app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450) 2022-12-05 17:18:09 -08:00
Aliaksandr Valialkin
f3e84b4dea
{dashboards,alerts}: subtitute {type="indexdb"} with {type=~"indexdb.*"} inside queries after 8189770c50
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 16:00:22 -08:00
Aliaksandr Valialkin
8189770c50
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:16:14 -08:00
Aliaksandr Valialkin
ddc3d6b5c3
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:17:31 -08:00
Aliaksandr Valialkin
c1cd4a9101
docs/CHANGELOG.md: consistently add - prefix in front of command-line flags
This is a follow-up for bcba5d2a78
2022-12-02 19:08:26 -08:00
Aliaksandr Valialkin
b6712ac08e
docs: follow-up after 30fea30685
- Run `make docs-sync`, so app/vmalert/README.md is copied to docs/vmalert.md
- Clarify the feature description in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3408
2022-12-02 19:03:11 -08:00
Denys Holius
54741f6f38
docs/Articles.md: fir broken link (#3433) 2022-12-02 10:35:40 +01:00
Roman Khavronenko
cd5c451ea3
docs: fix typo in cluster's README (#3430)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 16:24:16 +01:00
Roman Khavronenko
bcba5d2a78
vmalert: fix replay step param (#3428)
The recent change in modifying default value
of `datasource.queryStep` flag resulted in situation
where replay mode was always running queries with
step=`datasource.queryStep`. When it should always
use rule's evaluation interval.

The fix is related not to replay mode only, but
for all Range requests. Now step param is set
individually for each mode.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 13:57:53 +01:00
Zakhar Bessarab
30fea30685
app/vmalert: add remoteWrite.sendTimeout command-line flag to configure timeout for sending data to remoteWrite.url (#3423)
* app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url`

* vmalert: remove WriteTimeout from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the WriteTimeout.

* vmalert: remove DisablePathAppend from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the DisablePathAppend.

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:57:19 +01:00
Roman Khavronenko
8cc4f7eac6
vmalert: properly pass headers during the restore procedure (#3420)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:27:39 +01:00
Aliaksandr Valialkin
3cdff3de23
docs/vmagent.md: update after 959f06d175 2022-11-29 21:33:49 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
c7ce4979ec
all: follow-up after 05cf8a6ecc 2022-11-29 21:03:59 -08:00
Aliaksandr Valialkin
4822406b64
app/vmalert: substitute -datasource.disablePathAppend with -remoteRead.disablePathAppend in the description for -datasource.url command-line flag
This is a follow-up for 959f06d175
2022-11-29 20:36:41 -08:00
Dmytro Kozlov
05cf8a6ecc
vmctl: support of the remote read protocol (#3232)
vmctl: support of the remote read protocol

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 22:53:28 +01:00
Max Golionko
959f06d175
vmalert: flag reference update (#3415)
* flag reference update

there is no flag `-datasource.disablePathAppend` and datasource actually checking for `-remoteRead.disablePathAppend`

* update source for doc as well
2022-11-29 19:22:57 +01:00
Zakhar Bessarab
2f837c1b23
doc/operator: update formatting for backup section, add FAQ section (#3416)
* doc/operator: update formatting for backup section, add FAQ section

* doc/operator: address review feedback

* doc/operator: add note about difference between `VMRestore` and `VMBackupmanager` as init containers

* Update docs/operator/backups.MD

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-11-29 19:19:46 +01:00
Aliaksandr Valialkin
0002de937b
docs/CHANGELOG.md: document 027ab74efb
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402
2022-11-28 18:55:01 -08:00
Aliaksandr Valialkin
5c906beea2
docs/url-examples.md: allow linking to how to send OpenTSDB/Graphite data chapters 2022-11-28 18:42:49 -08:00
Aliaksandr Valialkin
654e94f420
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
2022-11-28 18:37:09 -08:00
Aliaksandr Valialkin
090343ff50
docs/Articles.md: add a link to slides about scaling to 100 million metrics per second 2022-11-28 16:41:14 -08:00
Aliaksandr Valialkin
ad105147dd
docs/CHANGELOG.md: cut v1.84.0 2022-11-25 19:53:29 -08:00
Aliaksandr Valialkin
e014467f42
docs/README.md: make docs-sync after 58d459e8a8 2022-11-25 16:55:47 -08:00
Aliaksandr Valialkin
58d459e8a8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:48:14 -08:00
Zakhar Bessarab
8b6d528fbd
{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348)
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* app/vmselect: fix error message

* {app/vmstorage,app/vmselect}: fix error messages

* app/vmselect: change log level for error handling

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 11:05:47 -08:00
Aliaksandr Valialkin
da13d36af9
app/vmselect/vmui: make vmui-update after eb772aa50e 2022-11-24 17:37:18 -08:00
Yury Molodov
eb772aa50e
vmui: improve table view (#3377)
* vmui: add compact table view (#3365)

* feat: add compact table view

* fix: add overflow table

* fix: change table styles

* vmui: compact table view

* Update docs/CHANGELOG.md

Co-authored-by: Michal Kralik <michal.kralik@percona.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-24 17:33:07 -08:00
Aliaksandr Valialkin
399ed9a3b9
docs/MetricsQL.md: document that histogram_share() accepts optional boundsLabel arg 2022-11-24 17:27:30 -08:00
Aliaksandr Valialkin
045fec631b
docs/vmagent.md: typo fix 2022-11-24 17:19:45 -08:00
Aliaksandr Valialkin
04bb2e14dd
docs/Articles.md: add a link to "The cost of scale in Prometheus ecosystem" talk 2022-11-23 00:11:51 +02:00
Yury Molodov
7dc2349913
vmui: add set up series custom limits (#3368)
* feat: add set up series custom limits

* feat: add button for show series without limits

* fix: resolve merge conflicts
2022-11-22 15:31:17 +02:00
Aliaksandr Valialkin
b1622ad63e
docs/Articles.md: add a link to https://www.youtube.com/watch?v=_zORxrgLtec (OSA Con 2022: Specifics of data analysis in Time Series Databases) 2022-11-22 01:08:21 +02:00
Aliaksandr Valialkin
9498f871e7
app/vminsert: add missing vm_relabel_config_* metrics after 03d88bc066
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
2022-11-22 00:47:49 +02:00
Roman Khavronenko
03d88bc066
vmagent: expose metrics for tracking config state (#3375)
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:38:43 +02:00
Yury Molodov
7d1b3e7e14
vmui: add copy button to row on Table view (#3363)
* feat: add copy button to row on Table view

* vmui: add copy button to row on Table view

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-22 00:31:33 +02:00
Yury Molodov
82d254af08
vmui: sticky tooltip (#3376)
* feat: add ability to make tooltip "sticky"

* vmui: add ability to make tooltip "sticky"
2022-11-22 00:26:53 +02:00
Aliaksandr Valialkin
ee1479bac6
docs/CHANGELOG.md: link to the related issue for range_normalize() function 2022-11-21 23:27:00 +02:00
Aliaksandr Valialkin
d9c3a2b605
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range
This may be useful for analyzing correlation between time series with different value ranges
2022-11-21 23:25:00 +02:00
Aliaksandr Valialkin
95f0266558
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
2022-11-21 22:32:05 +02:00
Aliaksandr Valialkin
05ed98c98b
app/vmselect/promql: allow using SI and IEC suffixes in numeric values inside queries
For example, 10Ki is equivalent to 10*1024, while 5.3M is equivalent to 5.3*1000*1000
2022-11-21 21:27:55 +02:00
Aliaksandr Valialkin
2c9e403d5f
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
2022-11-21 16:47:37 +02:00
Aliaksandr Valialkin
b796a0dc3f
app/vmselect/promql: optimize e1 op e2 when e1 returns an empty result
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
2022-11-21 16:09:10 +02:00
Roman Khavronenko
84742f229a
vmalert: add default list of alerting rules (#3373)
The default list of alerting rules contains the basic
rules for checking vmalert's health state and is recommended
to use for monitoring vmalert deployments.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-21 14:45:45 +01:00
Aliaksandr Valialkin
20d758e3e4
all: add a link to https://docs.victoriametrics.com/enterprise.html into description for enterprise flags 2022-11-21 15:42:01 +02:00
Aliaksandr Valialkin
cb1a621d63
app/{vminsert,vmselect}: add -storageNode.filter command-line flag for filtering the discovered storage nodes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353
2022-11-21 15:20:14 +02:00
Aliaksandr Valialkin
65b4e96a80
docs/Cluster-VictoriaMetrics.md: sync with cluster branch 2022-11-18 14:06:23 +02:00
Aliaksandr Valialkin
a061d33400
app/vmselect: clarify that it isnt recommended setting -replicationFactor at vmselect nodes even if the replication is enabled at vminsert nodes 2022-11-18 14:04:53 +02:00
Yury Molodov
519bd2af7b
vmui: add trace analyzer (#3310)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add page with trace analyzer

* fix: change detect trace data

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* fix: change descriptions on trace page

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* feat: add base components

* feat: add reset styles

* docs: add description about trace analyzer

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

* feat: add paste json for trace analyzer

* vmui: update trace analyzer docs

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 22:22:01 +02:00
Aliaksandr Valialkin
e79bfdf4b8
docs/CHANGELOG.md: document the fix for CPU usage spikes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-17 22:02:02 +02:00
Aliaksandr Valialkin
578bb58ea9
app/vmselect/vmui: make vmui-update after 51bfd1ab80 2022-11-17 18:54:55 +02:00
Yury Molodov
51bfd1ab80
vmui: add ability hide query (#3359)
* feat: add ability hide query

* fix: change logic hide query

* fix: remove console.log
2022-11-17 18:42:33 +02:00
Yury Molodov
bc8a782f74
vmui/refactor (#3298)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add base components

* feat: add reset styles

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 01:09:14 +02:00
Aliaksandr Valialkin
a260e2659e
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:40 +02:00
Aliaksandr Valialkin
c1a3192d8b
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:38:48 +02:00
Aliaksandr Valialkin
c3362e3db4
lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio
Reduce the default value of this flag from 1% to 0.2% after 71335e6024

This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:39:39 +02:00
Aliaksandr Valialkin
09b79d74a7
docs/CHANGELOG.md: document changes in v1.79.5 release 2022-11-11 01:27:45 +02:00
Aliaksandr Valialkin
bbe1a1472c
docs/CHANGELOG.md: cut v1.83.1 2022-11-10 14:06:57 +02:00
Aliaksandr Valialkin
1b9dff133a
docs/CHANGELOG.md: document the fix at 71335e6024 2022-11-10 13:46:51 +02:00
Aliaksandr Valialkin
b8839df32c
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:35:05 +02:00
Aliaksandr Valialkin
e17a1acf4a
docs/CHANGELOG.md: document 7130af7fd2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2814
2022-11-09 12:15:49 +02:00
Denys Holius
aebe21e2c8
guides/README.md: fix link to guide-delete-or-replace-metrics.html (#3331) 2022-11-09 12:00:46 +02:00
Aliaksandr Valialkin
c973aca617
app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket
This should prevent from panics when the list of discovered vmstorage nodes changes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
2022-11-09 11:51:10 +02:00
Roman Khavronenko
9f8bf524ad
bump go version to 1.19.3 (#3327)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-08 16:43:59 +01:00
Zakhar Bessarab
91dd79f40f
docs/operator: fix description for SA at VMClusterSpec (#3313) 2022-11-07 16:51:01 +01:00
Aliaksandr Valialkin
7fa5d043f5
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does
See https://github.com/prometheus/prometheus/pull/11482
2022-11-07 15:25:53 +02:00
Aliaksandr Valialkin
daa70e6560
lib/storage: follow-up for 790768f20b
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:04:08 +02:00
Aliaksandr Valialkin
23c79e2e49
docs/Single-server-VictoriaMetrics.md: mention about security certifications at Security chapter
This is a follow-up for 94bd49402e
2022-11-05 10:07:29 +02:00
Aliaksandr Valialkin
4ef5fe1317
docs/guides/guide-vmcluster-multiple-retention-setup.md: clarify docs after a75d85b11e 2022-11-05 10:02:48 +02:00
Artem Navoiev
94bd49402e
docs: Add link to security page from Readme (#3286)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-11-04 09:49:12 +01:00
Dmytro Kozlov
99d8fcb332
docs/operator: change VMAgentRemoteWriteSettings.MaxDiskUsagePerURL type from int32 to int64 (#3307)
* docs/operator: change VMAgentRemoteWriteSettings.MaxDiskUsagePerURL type from int32 to int64

* docs: operator updates api description

Co-authored-by: f41gh7 <nik@victoriametrics.com>
2022-11-04 01:12:57 +01:00
Artem Navoiev
a75d85b11e
update multi-tenancy guide. Add infromation about Enterprise and Rete… (#3285)
docs: update multi-tenancy guide

Add information about Enterprise and Retention Filters

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-11-01 15:19:09 +01:00
Aliaksandr Valialkin
df88832c86
docs/Single-server-VictoriaMetrics.md: follow-up after a1a97b9321
- Remove trailing whitespace at the end of lines
- Remove redundant sentence stating that time series matching the given selector will be deleted.
  It should be clear from the surrounding context.
2022-11-01 10:49:10 +02:00
Dmytro Kozlov
a1a97b9321
docs: clarify information about usage of the api/v1/admin/tsdb/delete_series API (#3287)
docs: clarify information about usage of the `api/v1/admin/tsdb/delete_series` API
2022-11-01 09:36:28 +01:00
Aliaksandr Valialkin
a1011931ac
docs/Release-Guide.md: instruct to update VictoriaMetrics version in deployment/docker/docker-compose*.yml files after creating new release
This is a follow-up for d1509f4559
2022-11-01 10:31:02 +02:00
Aliaksandr Valialkin
619b3c926d
docs/enterprise.md: mention that feature requests from enterprise customers are prioritized 2022-11-01 10:28:12 +02:00
Aliaksandr Valialkin
0f8f36de24
docs: typo fixes 2022-10-29 04:52:18 +03:00
Aliaksandr Valialkin
740f7ac5e0
docs/CHANGELOG.md: cut v1.83.0 2022-10-29 02:54:54 +03:00
Aliaksandr Valialkin
cdd3443806
app/vmbackupmanager: add functionality for automated restore from backup 2022-10-29 02:30:52 +03:00
Aliaksandr Valialkin
76e8888272
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set.
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.

While at it, properly handle some corner cases:

- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278

Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
2022-10-28 22:14:26 +03:00
Aliaksandr Valialkin
9aee303ca1
docs/Cluster-VictoriaMetrics.md: improve docs for dns+srv service discovery 2022-10-28 14:24:48 +03:00
Aliaksandr Valialkin
a72c5f76eb
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
2022-10-28 13:13:45 +03:00
Aliaksandr Valialkin
a018b1d75e
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-28 00:01:16 +03:00
Aliaksandr Valialkin
4bd0244599
Revert "vmalert: escape query params if external alert source defined (#3267)"
This reverts commit 00c838353d.

Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 .
Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana:

https://grafana.example.com/explore%3ForgId%3D1%26left%3D...

The next commit will contain the correct fix of the issue - the `quotesEscape` function must
properly escape the string, so it could be embedded into JSON string. This function must
properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary.
Actually, the next commit makes the `crlfEscape` function deprecated.
2022-10-27 22:30:27 +03:00
Dmytro Kozlov
00c838353d
vmalert: escape query params if external alert source defined (#3267)
vmalert: escape query args if external alert source defined
2022-10-26 10:00:14 -04:00
Aliaksandr Valialkin
518c340ae3
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:49:33 +03:00
Yury Molodov
eae6063450
fix: change step setting field (#3270)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-26 02:46:46 +03:00
Yury Molodov
bc7456841f
vmui: add responsive styles for small screens (#3256)
* fix: add responsive styles for small screens

* fix: correct additional settings margins

* docs/CHANGELOG.md: add responsive styles
2022-10-26 02:39:54 +03:00
Aliaksandr Valialkin
685433b8da
docs/Cluster-VictoriaMetrics.md: update docs about env vars usage in command-line flags
This is a follow-up for 02096e06d0
2022-10-26 01:54:15 +03:00
Aliaksandr Valialkin
02096e06d0
lib/envflag: allow referring environment variables in command-line flags 2022-10-26 01:52:05 +03:00
Aliaksandr Valialkin
5d82e7d64a
docs: run make docs-sync after 2ed3d49c26 2022-10-26 01:14:31 +03:00
Aliaksandr Valialkin
db8abd000e
docs/Makefile: fix docs-up Makefile command after 9ccd22c1f6 2022-10-25 17:53:24 +03:00
Denys Holius
9ccd22c1f6
Docs: add guide "How to delete and replace metrics" (#2829)
docs: add guide how to delete and replace metrics
2022-10-25 09:02:14 -04:00
Aliaksandr Valialkin
15849cb571
docs/guides/migrate-from-influx.md: properly display images at https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/guides/migrate-from-influx.md
This is a follow-up for 42375679db
2022-10-24 21:28:31 +03:00
Nguyen Van Duc
42375679db
Fix images not display on key concepts document (#3266) 2022-10-24 21:22:41 +03:00
Aliaksandr Valialkin
b1324631b1
docs/CHANGELOG.md: document 274e235bf7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240
2022-10-24 21:02:17 +03:00
Aliaksandr Valialkin
c52c23c272
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:02:03 +03:00
Aliaksandr Valialkin
cac28ae0ae
docs/CHANGELOG.md: typo fixes 2022-10-24 16:59:55 +03:00
Aliaksandr Valialkin
8e998aa1a1
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:40:20 +03:00
Aliaksandr Valialkin
d7e657e5f9
docs/CHANGELOG.md: document 69b27275d2b2bf1bdae0d8b887b44bde2a787649 2022-10-24 16:08:15 +03:00
Aliaksandr Valialkin
5e4dfe50c6
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:23:47 +03:00
Aliaksandr Valialkin
a10647e0bf
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:13:47 +03:00
Aliaksandr Valialkin
4cd173bbaa
lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal
Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals
if they hang for some reason in graceful shutdown procedure.
2022-10-20 21:40:20 +03:00
Aliaksandr Valialkin
cba9696a14
docs/CHANGELOG.md: move the BUGFIX line for 1059c4d84a into correct place 2022-10-18 20:38:57 +03:00
Nikolay
1059c4d84a
lib/promscrape/discovery/kubernetes: correctly wrap error (#3250)
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-18 20:37:42 +03:00
Roman Khavronenko
4e0ea95f26
vmalert: lower severity level for RW retries (#3237)
The message about dropped data still remains at `error` level.
The change supposed to make log message more clear about how
serious it is.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-18 14:27:20 +02:00
Aliaksandr Valialkin
069401a304
all: log error when environment variables referred from -promscrape.config are missing
This should prevent from using incorrect config files
2022-10-18 10:47:16 +03:00
Aliaksandr Valialkin
646fb17237
docs/MetricsQL.md: document that input histograms must have the same set of buckets when calculating the quantile over them
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3231
2022-10-14 13:59:58 +03:00
Aliaksandr Valialkin
adf3419699
docs/CHANGELOG.md: add a note that it is recommended to use v1.82.1 instead of v1.82.0 2022-10-14 13:41:01 +03:00
Aliaksandr Valialkin
3a0c69651a
docs/CHANGELOG.md: release v1.82.1 2022-10-14 11:33:25 +03:00
Aliaksandr Valialkin
2e3be68617
lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map
This prevents from possible corruption of the internal cache map
when the underlying byte slice used by the string key is modified.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227
2022-10-14 09:51:19 +03:00
Denys Holius
b1de74a6ba
k8s-monitoring-via-vm-cluster.md: remove extra spaces (#3234) 2022-10-13 17:25:35 +02:00
Yury Molodov
ff6151fa49
vmui: limit number of plotted series (#3229)
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:13:47 +03:00
Aliaksandr Valialkin
e6fd33044f
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:03:08 +03:00
Aliaksandr Valialkin
76e275ddef
docs/CHANGELOG.md: document b856581ad3 2022-10-13 10:35:48 +03:00
Dan Fredell
42ce4364fc
Multi retention standardization of docs (#3223)
* Multi retention standardization of docs

While reading through the docs the implementation details had different formatting for storageNode. This standardizes them and adds a link to the retention docs.

* Update docs/guides/guide-vmcluster-multiple-retention-setup.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2022-10-13 10:27:16 +03:00
Roman Khavronenko
96a106eab2
vmalert: update troubleshooting docs (#3228)
The default value of `-datasource.queryStep` has changed, so we update
the troubleshooting docs accordingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-12 12:12:37 +02:00
Aliaksandr Valialkin
cfae887c75
docs/CHANGELOG.md: document a4975ace86
The original commit, which led to the issue - 877940a131
2022-10-12 09:30:36 +03:00
Aliaksandr Valialkin
41925a9500
docs/CHANGELOG.md: document 9544b5cacf7446203fac19eb7c575779dc9b280e 2022-10-12 09:25:17 +03:00
Aliaksandr Valialkin
a79272db1f
docs/vmbackup.md: run make docs-sync after 076e721c22 2022-10-10 21:57:46 +03:00
Aliaksandr Valialkin
875abf0ef4
docs/CHANGELOG.md: document e384d88abf 2022-10-10 21:52:06 +03:00
Aliaksandr Valialkin
04a05f161c
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-10 21:45:13 +03:00
Aliaksandr Valialkin
921918cb49
docs/sd_configs.md: document __scrape_timeout__, __scrape_interval__ and __series_limit__ labels 2022-10-09 15:05:30 +03:00
Aliaksandr Valialkin
50f5eae0e0
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.

This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
2022-10-09 14:51:16 +03:00
Aliaksandr Valialkin
8e1ccecd97
docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag
This is a follow-up for 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-09 13:56:59 +03:00
Aliaksandr Valialkin
4f4d591ccb
docs/vmbackupmanager.md: update docs after adding the support to make backups to Azure blob storage
This is a follow-up for 262ce77e2d
2022-10-08 10:30:39 +03:00
Aliaksandr Valialkin
8f8ce5e238
docs: add description for -search.maxMemoryPerQuery command-line flag 2022-10-08 01:16:55 +03:00
Aliaksandr Valialkin
5138eaeea0
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-08 01:08:05 +03:00
Aliaksandr Valialkin
3aafbc3624
docs/CHANGELOG.md: add a link to a feature request for the feature, which allows specifying full scrape urls in targets list and in the __address__ label
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3208
2022-10-07 23:45:57 +03:00
Aliaksandr Valialkin
5269b1ad77
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
2022-10-07 23:36:14 +03:00
Aliaksandr Valialkin
93811da76d
docs/CHANGELOG.md: document the 27ed4b853e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205
2022-10-07 23:06:32 +03:00
Aliaksandr Valialkin
b47caa86db
all: update the minimum required Go verson from 1.19.1 to 1.19.2
This is needed because of security vulnerabilities found in Go 1.19.1
See https://go.dev/doc/devel/release#go1.19.2
2022-10-07 22:43:37 +03:00
Aliaksandr Valialkin
f9df0cae16
lib/promscrape: allow specifying full target url in __address__ label
Previously the `__address__` label could contain only `host:port` part of the target url,
while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__`
labels. Now it is possible to set the full url in `__address__` label.

This makes valid the following scrape config, which is frequently used by novice users:

scrape_configs:
- job_name: foo
  static_configs:
  - targets:
    - http://host1/metrics1
    - https://host2/metrics2
2022-10-07 22:43:04 +03:00
Aliaksandr Valialkin
8bc840358f
docs/CHANGELOG.md: cut v1.82.0 2022-10-07 03:15:23 +03:00
Aliaksandr Valialkin
7b6ce3f75e
docs/CHANGELOG.md: typo fix 2022-10-07 03:11:22 +03:00
Aliaksandr Valialkin
ba4050ab1f
docs/CHANGELOG.md: add a changelog for v1.79.4 LTS release (copied from lts-1.79 branch) 2022-10-07 02:50:59 +03:00
Aliaksandr Valialkin
285e92706d
docs/MetricsQL.md: formatting improvements
Also put a link to the function type in docs for every function.
This should simplify understanding of MetricsQL functions for novice users.
2022-10-07 00:53:14 +03:00
Aliaksandr Valialkin
f452c84579
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:22:24 +03:00
Aliaksandr Valialkin
40e899fd67
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:21 +03:00
Aliaksandr Valialkin
8ae713253e
docs/CHANGELOG.md: remove duplicate description of the bugfix, which has been included in v1.81.2 2022-10-06 15:53:05 +03:00
Roman Khavronenko
5b10fa87b2
vmalert: fix misleading line regarding multitenancy (#3206)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 15:03:45 +03:00
Aliaksandr Valialkin
8d7910a463
docs/CHANGELOG.md: add missing link to /api/v1/export docs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 15:02:13 +03:00
Aliaksandr Valialkin
d9282027e6
app: follow-up after ec04fcac93
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:52:02 +03:00
Dmytro Kozlov
ec04fcac93
Properly parse json when export import metric (#3180)
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API

* app/vmselect: update convert function

* app/vmselect: export null if `math.IsNaN(v)`

* app/vmselect: get float from json

* lib/protoparser: add test

* docs: add change log

* lib/protoparser: make export import api compatible
2022-10-06 13:54:20 +03:00
Aliaksandr Valialkin
dba49943d3
docs/Single-server-VictoriaMetrics.md: update docs after the a54987f671 2022-10-06 13:29:38 +03:00
Yury Molodov
a54987f671
vmui: maximum queries (#3196)
vmui: allow using up 4 queries at the same time

The change also introduces UI updates to make using 
multiple queries more conveniently.
2022-10-06 07:10:21 +02:00
Aliaksandr Valialkin
5307cf068f
docs: follow-up after 262ce77e2d
* Document the addition of Azure blob storage support in vmbackup / vmrestore
* List the supported storage system types at docs/vmrestore.md
* Mention about azblob storage system support at -src and -dst command-line flags
  for vmbackup / vmrestore tools.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029
2022-10-06 00:35:34 +03:00
Aliaksandr Valialkin
f596e49881
docs/CHANGELOG.md: document f8ac55d70ada9ef8490b322abefb05f28f75e2e9 2022-10-06 00:05:37 +03:00
Aliaksandr Valialkin
c45c61cf93
app/vmalert: follow-up after f8ac55d70ada9ef8490b322abefb05f28f75e2e9
* Use vm_account_id and vm_project_id labels to be consistent with https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels
* Document the feature that vmalert now exposes vm_account_id and vm_project_id
  labels if -clusterMode is set.
* Use literal strings instead of string constants for vm_account_id and vm_project_id.
  This improves code readability.
2022-10-06 00:05:33 +03:00
Aliaksandr Valialkin
703094a37a
app/vmbackupmanager: expose vm_backup_in_flight metrics (follow-up after c6bca4a0b47b4f5626e1913d3480e62d657ed4cf) 2022-10-05 23:30:21 +03:00
Aliaksandr Valialkin
f9fc838b7b
app/vmalert: update -external.alert.source command-line flag description after 61544e13ad 2022-10-05 22:52:38 +03:00
Aliaksandr Valialkin
440495df52
Makefile: fix sed command and if condition for Linux bash after 2d11896486 2022-10-05 22:40:04 +03:00
Roman Khavronenko
61544e13ad
vmalert: allow using {{$labels}} for templating in -external.alert.source (#3194)
The change is supposed to provide additional flexibility for generating alert's
source link based on label values.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-05 19:25:03 +02:00
Dmytro Kozlov
e0ea76db62
docs: add cardinality explorer blog post (#3198)
docs: add cardinality explorer blog post
2022-10-05 13:19:41 +02:00
Roman Khavronenko
2d11896486
docs: udpate Datadog section (#3190)
The `docs-sync` command was updated to modify images path
for assets in `docs/` folder. The change allows to refer images from `docs`
without copying them to the root folder.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-03 13:13:07 +02:00
Yurii Kravets
272f00dbb6
docs: Update How to send data from DataDog agent (#3168)
docs: Update How to send data from DataDog agent
2022-10-03 11:45:05 +02:00
Aliaksandr Valialkin
c53b7e66ef
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:05:43 +03:00
Aliaksandr Valialkin
fcc7ab71b3
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-01 19:47:37 +03:00
Aliaksandr Valialkin
7812761ab4
docs/Cluster-VictoriaMetrics.md: add a security note for multitenant support via labels 2022-10-01 18:57:28 +03:00
Aliaksandr Valialkin
d7327d2f02
docs/vmagent.md: fix incorrect url for multitenant writes to VictoriaMetrics cluster 2022-10-01 18:51:37 +03:00
Aliaksandr Valialkin
0dc93cca7f
app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
2022-10-01 18:40:59 +03:00
Aliaksandr Valialkin
a296994fed
app/vmauth: do not remove trailing slash from the proxied path
This should fix the issue with opening VMUI at /vmui/ page.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:52:30 +03:00
Aliaksandr Valialkin
4998402004
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin
973ce4b561
app/vmagent: accept requests with /prometheus and /influx path prefixes in the same way as VictoriaMetrics does
This allows using vmagent as a drop-in replacement for VictoriaMetrics
for push protocols.
2022-10-01 14:10:48 +03:00
Aliaksandr Valialkin
63a94b1d54
docs/vmagent.md: formatting fixes 2022-10-01 13:36:24 +03:00
lionelee
1fda517af9
docs/vmagent.md: fix typo (#3186) 2022-10-01 12:30:01 +03:00
Aliaksandr Valialkin
db16759c68
lib/storage: optimize matching speed for non-trivial regexp filters
Wrap re.Match into bytesutil.FastStringMatcher.

This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:06:06 +03:00
Aliaksandr Valialkin
9e8fbef27e
docs/CHANGELOG.md: clarify the description of the improvement in relabeling performance 2022-10-01 11:54:48 +03:00
Aliaksandr Valialkin
c54e14cdec
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:48:32 +03:00
Aliaksandr Valialkin
d0b7172316
app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints
These labels are substituted by documented vm_account_id and vm_project_id labels.

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels

This is a follow up for 505d359b39
2022-09-30 18:35:53 +03:00
Nikolay
33f40f4a5f
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 18:35:53 +03:00
Roman Khavronenko
740bb2cc00
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 17:10:17 +02:00
Aliaksandr Valialkin
a18d6d5ccc
lib/promrelabel: optimize action: replace for non-trivial regex values
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.

Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:25:05 +03:00
Dmytro Kozlov
e220bc3cd5
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:55:12 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:20:34 +03:00
Aliaksandr Valialkin
fda60b3d4d
lib/promrelabel: properly parse regex with escaped $ at the end
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131

Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin
bf2f14a3a6
docs/CHANGELOG.md: document 39f559d22b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013
2022-09-30 07:48:25 +03:00
Nikolay
f61b8cec69
lib/awsapi: fixes sign encoding (#3183)
* lib/awsapi: fixes sign encoding

previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171

* Update lib/awsapi/sign.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:43:44 +03:00
nemobis
6a818dbddf
docs: fix typo in Cluster-VictoriaMetrics.md (#3172) 2022-09-28 09:18:29 +02:00
panguicai
fbc85e654c
docs: fix typo for vmalert docs (#3173)
Signed-off-by: panguicai008 <1121906548@qq.com>
2022-09-28 09:15:05 +02:00
Roman Khavronenko
4ad3b36630
docs: update img for key concepts (#3176)
Make dots bigger for range_query, so it will be easier
to read the graph.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-27 21:22:59 +02:00
Dmytro Kozlov
6ab1cede62
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:35:45 +03:00
Denys Holius
d63410bf6f
docs: managed_victoriametrics/quickstart.md fix image link (#3165) 2022-09-26 16:35:18 +02:00
Roman Khavronenko
36a9a834b3
docs: clarify numeric label values for label_value (#3129)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3114
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 16:48:38 +03:00
Max Golionko
d67948b8e3
added security policy (#3140)
* added security policy

* Update SECURITY.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:47:02 +03:00
Roman Khavronenko
e96ccf3f71
lib/mergeset: follow-up after a0e7432e42 (#3145)
* lib/mergeset: follow-up after a0e7432e42

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:39:56 +03:00
Aliaksandr Valialkin
72c29d762e
docs/CHANGELOG.md: document f022296d96
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147
2022-09-26 16:32:19 +03:00
Aliaksandr Valialkin
a2431c2a88
docs/CHANGELOG.md: document 166d444159 2022-09-26 15:39:14 +03:00
Aliaksandr Valialkin
41f8c2987d
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102

See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:17:25 +03:00
Aliaksandr Valialkin
5983ecf4d1
docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics sanitizes metric names in DataDog data by default
This is a follow-up for 7c2474dac7
2022-09-26 14:02:43 +03:00
Aliaksandr Valialkin
7c2474dac7
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:57:23 +03:00
Aliaksandr Valialkin
fcffdba9dc
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:11:39 +03:00
Aliaksandr Valialkin
819aa95552
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-09-26 08:47:31 +03:00
Aliaksandr Valialkin
6b71f33c8b
docs/vmalert.md: follow-up after 7748a9d629
- Consistently use single dash in front of command-line flags instead of double dashes.
- Add a warning that too small -search.latencyOffset may lead to incomplete query results.
2022-09-26 08:36:24 +03:00
Aliaksandr Valialkin
b68cd810fc
docs: follow-up for 03d54ac890
- Typo fix: 'adn' -> 'and'
- Remove duplicate link to https://victoriametrics.com/blog/victoriametrics-monitoring/
  from docs/Articles.md . The initial link has been already added in 27254096b2
2022-09-26 08:26:38 +03:00
Roman Khavronenko
0c95f928ae
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-23 10:22:53 +02:00
Roman Khavronenko
7748a9d629
vmalert: add info about search.latencyOffset to Troubleshooting (#3151)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-23 09:49:14 +02:00
Roman Khavronenko
03d54ac890
docs: mention VictoriaMetrics Monitoring blog post (#3152)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-23 07:12:15 +02:00
Aliaksandr Valialkin
27254096b2
docs/Articles.md: add a link to https://victoriametrics.com/blog/victoriametrics-monitoring/ 2022-09-22 19:17:10 +03:00
Aliaksandr Valialkin
a3e536c0e7
docs/Articles.md: add a link to https://dataswamp.org/~solene/2022-09-11-exploring-monitoring-stacks.html 2022-09-22 19:17:10 +03:00
Roman Khavronenko
5714a68ac6
deployment/docker: move cluster compose env to master branch (#3130)
* deployment/docker: move cluster compose env to master branch

The change supposed to simplify the process of maintaining for
single/cluster docker-compose envs, alerts, dashboards. It also
supposes to reduce confusion for users when looking for cluster
related alerts/configs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/docker: move cluster compose env to master branch

Review updates.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-21 11:48:38 +03:00
Aliaksandr Valialkin
1aef635de4
docs/CHANGELOG.md: clarify the change at 622bbedbe1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119
2022-09-19 15:02:05 +03:00
Aliaksandr Valialkin
584a5d6b1f
docs/managed_victoriametrics/quickstart.md: small fixes after 606166ef68 2022-09-19 14:52:46 +03:00
Roman Khavronenko
622bbedbe1
vmalert: always re-evaluate Annotations (#3119)
* vmalert: always re-evaluate Annotations

Previously, Annotations were evaluated only:
1. On alert creating.
2. On alert's value change.

This is premature optimization. It was assumed that since annotations
could contain only text with alert's labels or value - there is no need
in spending resources to re-compile Annotations.

Later, template function `query` was added, which can execute
arbitrary queries and return different results on every evaluation.
So if it was used in annotations, it would be executed only on init
or value change.

Another case when optimization caused an issue - annotations hot reload.
In this case, annotations of the active alert won't change even if Rule's
annotations were changed.

This fix enables Annotations re-evaluation on each iteration to resolve
issues above. It would have some impact on performance, but it is unlikely
it will be noticeable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add tp Changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-16 16:19:10 +02:00
Roman Khavronenko
3484673566
vmalert: add Troubleshooting section to docs (#3115)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-15 16:15:39 +02:00
Dmytro Kozlov
606166ef68
docs/managed_victoriametrics: add how to restore password section (#3116)
docs/managed_victoriametrics: add how to restore password section
2022-09-15 16:02:31 +02:00
Aliaksandr Valialkin
455002922e
docs/Cluster-VictoriaMetrics.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:22:58 +03:00
Aliaksandr Valialkin
e7995375b5
docs: update -help output after explicit mentioning of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
4193af4571
docs/vmauth.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
3b2599c659
docs/vmalert.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:22:57 +03:00
Aliaksandr Valialkin
bbecd27557
docs: update docs with explicitly marked enterprise command-line flags for VictoriaMetrics and vmagent 2022-09-15 13:22:56 +03:00
Dmytro Kozlov
a9629cc32d
app/vmctl: add description about influx-skip-database-label flag (#3111) 2022-09-15 10:46:54 +02:00
Aliaksandr Valialkin
b869c757a9
docs/FAQ.md: add an answer for What is the difference between single-node and cluster versions of VictoriaMetrics? 2022-09-15 09:56:41 +03:00
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:41:09 +03:00
Aliaksandr Valialkin
5306f79fd1
docs/Articles.md: add a link to https://www.groundcover.com/blog/prometheus-grafana-kubernetes 2022-09-14 15:11:19 +03:00
Roman Khavronenko
877940a131
vmalert: add experimental feature of storing Rule's evaluation state (#3106)
vmalert: add experimental feature of storing Rule's evaluation state

The new feature keeps last 20 state changes of each Rule
in memory. The state are available for view on the Rule's
view page. The page can be opened by clicking on `Details`
link next to Rule's name on the `/groups` page.

States change suppose to help in investigating cases when Rule
doesn't generate alerts or records.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 14:04:24 +02:00
Roman Khavronenko
efea51a9ee
bump Go version to 1.19.1 (#3108)
The reason is to cover vulnerability GO-2022-0969
Found in: net/http@go1.18.5
Fixed in: net/http@go1.19.1
More info: https://pkg.go.dev/vuln/GO-2022-0969

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 12:29:19 +02:00
Aliaksandr Valialkin
74c00a8762
lib/promscrape: read response body into memory in stream parsing mode before parsing it
This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin
3af24a5b7c
docs: consistently use docs.victoriametrics.com instead of victoriametrics.github.io in all the links 2022-09-13 16:50:13 +03:00
Aliaksandr Valialkin
0ead64b6cf
app/vmalert: follow-up after 8441375da2
- Rename logDebug() to logDebugf() and pass format string together
  with format args directly to logDebugf(). This eliminates fmt.Sprintf()
  overhead at logDebug() call site when debugging is disabled.

- Format labels in debug message in Prometheus format, e.g. {label1="value1",...labelN="valueN"}

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025
2022-09-13 16:35:14 +03:00
Roman Khavronenko
8441375da2
vmalert: add debug mode for alerting rules (#3055)
* vmalert: add `debug` mode for alerting rules

Debug information includes alerts state changes and requests
sent to the datasource. Debug can be enabled only on rule's
level. It might be useful for debugging unexpected
behaviour of alerting rule.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmalert/alerting.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

* vmalert: go fmt

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-13 16:25:43 +03:00
Aliaksandr Valialkin
25c9a1604a
docs/CHANGELOG.md: document atomic directory deletion
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:39 +03:00
Aliaksandr Valialkin
db4f0fe6fc
docs/ExtendedPromQL.md: move the page to the bottom of the contents list at http://docs.victoriametrics.com 2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin
e041c913bd
Revert "docs/ExtendedPromQL.md: remove outdated doc"
This reverts commit 971e3d83f7.

Reason for revert: the ExtendedPromQL doc is still referred from third-party sites.
2022-09-13 16:17:37 +03:00
Craig Rodrigues
57ea8dbb36
docs: Use headings in DataDog section to make things easier to read (#3089)
Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>

Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>
2022-09-12 10:58:25 +03:00
Yury Molodov
1b41169415
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:55:11 +03:00
Aliaksandr Valialkin
c48ff746c6
docs/FAQ.md: add a link to https://victoriametrics.com/blog/mimir-benchmark/ to VictoriaMetrics vs Mimir chapter 2022-09-12 10:37:28 +03:00
Aliaksandr Valialkin
c4af0e833a
docs/Articles.md: add a link to https://victoriametrics.com/blog/mimir-benchmark/ 2022-09-11 15:23:35 +03:00
Yury Molodov
9541ef2e9e
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:43:37 +03:00
John Belmonte
defced2599
MetricsQL doc spellcheck (#3080) 2022-09-08 21:28:03 +03:00
Aliaksandr Valialkin
53b0c2eee4
docs/CHANGELOG.md: document ec273eafef
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3076
2022-09-08 21:25:30 +03:00
Aliaksandr Valialkin
b9231c715a
docs/Release-Guide.md: require manual push of the created release tags to public Github
The automated push of release tags to Github require specifying the remote repository name
when doing `git push <remote-repo-name> v1.xx.y`.
The remote repository name can differ in different environments,
so it cannot be put into Makefile rule.

TODO: create a Makefile rule, which generates standard remote names for public
and private repositories in Git, so `git push` for release tags could be automated then.
2022-09-08 15:00:18 +03:00
Aliaksandr Valialkin
07441b1cee
app/vmselect/netstorage: fix a typo, which leads to incorrect query results in VictoriaMetrics cluster
The typo has been introduced in the commit 1a254ea20c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067
2022-09-08 13:48:20 +03:00
Aliaksandr Valialkin
9f20d01a81
Revert "docs: add Monitoring at scale with Victoria Metrics (#3078)"
This reverts commit f24572fa65,
because the article https://tech.bedrockstreaming.com/2022/09/06/monitoring-at-scale-with-victoriametrics.html
has been already added at 09ff3f1928
2022-09-08 11:05:25 +03:00
Roman Khavronenko
f24572fa65
docs: add Monitoring at scale with Victoria Metrics (#3078)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-06 16:29:48 +02:00
Max Golionko
7da9443686
moved cluster dashboard to master (#3074)
dashboards: move cluster dashboard to master branch

This change should simplify dashboards management.
2022-09-06 16:19:43 +02:00
Aliaksandr Valialkin
ef7fdbb63c
docs/Single-server-VictoriaMetrics.md: add a note that cardinality explorer at cluster version of VictoriaMetrics may return lower than expected number of unique label values
See the corresponding comment in the code:

5a6e617b5e/app/vmselect/netstorage/netstorage.go (L1039-L1045)
2022-09-06 14:46:35 +03:00
Aliaksandr Valialkin
09ff3f1928
docs/Articles.md: add https://tech.bedrockstreaming.com/2022/09/06/monitoring-at-scale-with-victoriametrics.html 2022-09-06 13:32:41 +03:00
Aliaksandr Valialkin
651ace6ce4
docs/vmctl.md: make docs-sync after c5261d5f56
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:19:48 +03:00
Aliaksandr Valialkin
5fa9525498
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
This should reduce chances of unnoticed on-disk data corruption.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011

This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.

This is OK, since the native export format is undocumented.
2022-09-06 13:08:09 +03:00
Zakhar Bessarab
c5261d5f56
vmctl: implement support of chunking data for vm-native export (#3044)
vmctl: implement support of chunking data for vm-native export process

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 09:09:34 +02:00
Craig Rodrigues
462fc7b394
docs: Clarify DataDog examples for VictoriaMetrics cluster (#3048)
Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>
2022-09-05 16:25:05 +02:00
Yurii Kravets
7b04112352
docs: Update keyConcepts (#3040)
docs: update keyConcepts

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-09-05 13:59:28 +02:00
Aliaksandr Valialkin
ae31b2363f
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:55:48 +03:00
Aliaksandr Valialkin
7dc632719d
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin
ac98ecdc1d
docs/CHANGELOG.md: cut v1.81.1 2022-09-02 21:58:28 +03:00
Aliaksandr Valialkin
08538ff82a
app/vmselect/netstorage: fix potential panic under high load
The panic may trigger during data blocks' processing received
from vmstorage nodes when some of vmstorage nodes return an error
or when `-replicationFactor` is set to values higher than 2 at `vmselect`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058
2022-09-02 21:38:27 +03:00
Aliaksandr Valialkin
4076277cf0
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 19:46:27 +03:00
Aliaksandr Valialkin
f9d4ade35a
docs/Articles.md: add a link to https://aatarasoff.medium.com/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b 2022-08-31 05:03:22 +03:00
Aliaksandr Valialkin
b26de84b4a
docs/FAQ.md: mention that single-node vm is easier to setup and operate than the cluster version of vm 2022-08-31 05:02:15 +03:00
Aliaksandr Valialkin
6b0050a028
docs/CHANGELOG.md: cut v1.81.0 2022-08-31 02:33:23 +03:00
Max Golionko
c685afebb2
simplify release process (#3012)
* simplify release process

* address comments

* address comments

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-31 02:27:24 +03:00
Aliaksandr Valialkin
6c5d5bc7e6
docs/CHANGELOG.md: add missing link to vmagent 2022-08-31 02:21:44 +03:00
Aliaksandr Valialkin
c84e6429a0
docs/CHANGELOG.md: document v1.79.3 LTS release 2022-08-31 00:22:31 +03:00
Aliaksandr Valialkin
022bb62fa1
docs: clarify why cluster version is recommended to use for serving high loads 2022-08-31 00:22:30 +03:00
Craig Rodrigues
a54d7c24ff
docs: Add DataDog to list of ingestion methods (#3047)
Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>

Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>
2022-08-31 00:21:53 +03:00
Craig Rodrigues
e68a97f8d4
docs: add https://github.com/VictoriaMetrics/ansible-playbooks to README.md (#3046)
Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2022-08-30 20:52:24 +02:00
Aliaksandr Valialkin
bcdba8be18
docs/FAQ.md: add chapters explaining main differences between single-node and cluster versions of VictoriaMetrics
- Which type of VictoriaMetrics is better to use in production?
- How to migrate data from single-node VictoriaMetrics to cluster?
2022-08-30 13:12:03 +03:00
Artem Navoiev
5d4b1bc742
update multi-region guide, specify multi-level vmselect option (#3039)
* update multi-region guide, specify multi-level vmselect option

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* add more references

* Apply suggestions from code review

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-30 12:26:23 +03:00
Aliaksandr Valialkin
1eec2460ba
docs: fix links to DogStatsD
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3042
2022-08-30 10:49:26 +03:00
Aliaksandr Valialkin
d2e94ee91a
docs/CHANGELOG.md: document the 044d51b668 2022-08-30 09:42:19 +03:00
Aliaksandr Valialkin
ad11b8d83d
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:35:49 +03:00
Dmytro Kozlov
2d71b4859c
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:33:56 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName()
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
7afe8450fc
lib/promrelabel: optimize matching for commonly used regex patterns in if option
The following regex patterns are optimized:

- literal string match, e.g. "foo"
- prefix match, e.g. "foo.*" and "foo.+"
- substring match, e.g. ".*foo.*" and ".+foo.+"
- alternate values match, e.g. "foo|bar|baz"
2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin
0d4ea03a73
lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values
For example, the following relabeling rule must work much faster now:

- action: labeldrop
  regex: "foo|bar|baz"
2022-08-24 17:54:29 +03:00
Aliaksandr Valialkin
cdffe401e4
app/vmagent: follow-up after 2b22aa1537
- Document the change at docs/CHANGELOG.md
- Move auth token parsing from app/vmagent/opentsdbhttp/ to app/vmagent/main.go,
  since it must be parsed only when multitenancy support is enabled at vmagent side.
  See https://docs.victoriametrics.com/vmagent.html#multitenancy
2022-08-24 16:18:59 +03:00
Dmytro Kozlov
463ea6897b
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:25:18 +03:00
Aliaksandr Valialkin
3d12ee47f9
docs: mention that it is safe sharing the collected profiles from security PoV
The collected profiles do not contain sensitive information
2022-08-24 14:07:36 +03:00
Aliaksandr Valialkin
796aa310c2
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:

    vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:44:04 +03:00
Aliaksandr Valialkin
b1e1c50627
all: update Google Analytics tracking code from Unversal Analytics to v4
This is needed because Google Analytics devs decided to force their users
to update all their tracking codes to GA v4.

See https://support.google.com/analytics/answer/9744165

The link to the new tracking property ( G-N9SVT8S3HK ) - https://analytics.google.com/analytics/web/?authuser=1#/a129683199p328513681/admin/streams/table/4004610168
2022-08-24 12:16:37 +03:00