Commit Graph

370 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
af366a1f97
README.md: sync with docs/Cluster-VictoriaMetrics.md after e0569a355b 2024-02-23 00:22:54 +02:00
Aliaksandr Valialkin
f5680a6857
all: upgrade Go builder from Go1.21.7 to Go1.22.0
See https://go.dev/doc/go1.22
2024-02-12 22:14:00 +02:00
Aliaksandr Valialkin
46de5d7eb1
docs: update docs after ae8a867924
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470
2024-02-09 04:20:09 +02:00
Aliaksandr Valialkin
202d8e2c40
docs: update -help output after 61d9df4c36
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834
2024-02-08 14:50:56 +02:00
Aliaksandr Valialkin
898414856f
docs/Cluster-VictoriaMetrics.md: document the /datadog/api/beta/sketches endpoint
This is a follow-up for a1d1ccd6f2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584
2024-02-07 01:36:08 +02:00
Aliaksandr Valialkin
bace92fab6
lib/httpserver: add support for mTLS for requests to -httpListenAddr 2024-02-06 17:47:27 +02:00
Aliaksandr Valialkin
01312003be
docs/Cluster-VictoriaMetrics.md: add a warning that -disableReroutingOnUnavailable command-line flag may result in long pause for data ingestion when some of vmstorage nodes are unavailable for long time 2024-02-06 16:01:07 +02:00
Aliaksandr Valialkin
63a43331a3
docs/Cluster-VictoriaMetrics.md: document -disableReroutingOnUnavailable command-line flag
This is a follow-up for 88f0d1572e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713
2024-02-05 15:17:09 +02:00
Aliaksandr Valialkin
e2c58a9db1
docs/Cluster-VictoriaMetrics.md: mention -metrics.exposeMetadata command-line flag in Monitoring section 2024-02-05 11:48:05 +02:00
Aliaksandr Valialkin
eff4dc3685
docs: sync -help output after recent changes 2024-02-01 15:06:19 +02:00
hagen1778
216f268c1a
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 17:06:26 +01:00
Artem Navoiev
0d210308c3
docs: remove witdh from images, remove <p>, remove <div> (#5705)
* docs: remove witdh from images, remove <p>, remove <div>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* docs: remove <div> clarify language in code blocks

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-29 16:50:51 +01:00
Artem Navoiev
d42908133c
docs: remove <p> for imanges (#5702)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 22:34:40 +01:00
Artem Navoiev
36fa314161
remove all <div> as far they obsolete and can break markdown (#5701)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 22:32:54 +01:00
Aliaksandr Valialkin
2b0123058a
docs: update -help output after bb7a419cc3 2024-01-26 22:29:22 +01:00
Aliaksandr Valialkin
0dca3c4025
app/{vmselect,vmstorage}: return compression of the data passed from vmstorage to vmselect
This reverts cd4f641d32 , since it has been appeared that the disabled compression
for vmstorage->vmselect data increase network bandwidth usage by more than 10x on typical production workloads,
while it decreases CPU usage at vmstorage by up to 10% and improves query latency by up to 10%.

The 10x increase in network usage is too high price for 10% improvements on query latency and vmstorage CPU usage.
This may result in network bandwidth bottlenecks, which can reduce the overall performance and stability
of VictoriaMetrics cluster. That's why return back the vmstorage->vmselect data compression by default.

The vmstorage->vmselect compression can be disabled by passing -rpc.disableCompression command-line flag to vmstorage.
The vmselect->vmselect compression in multi-level cluster setup can be disabled by passing -clusternative.disableCompression command-line flag.
2024-01-24 13:37:05 +02:00
Aliaksandr Valialkin
5205f1c6a6
docs/Cluster-VictoriaMetrics.md: document that vmstorage doesnt compress data it sends to vmselect by default
This is a follow-up for cd4f641d32
2024-01-23 23:21:31 +02:00
Yury Molodov
3a26e4d6ec
vmui: add flag for default timezone setting (#5611)
* vmui: add flag for default timezone setting #5375

* vmui: validate timezone before client return

* Update app/vmselect/vmui.go

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 04:15:14 +02:00
Aliaksandr Valialkin
cd4f641d32
app/{vmstorage,vmselect}: disable vmstorage->vmselect RPC compression by default in order to improve query performance 2024-01-23 02:29:13 +02:00
Aliaksandr Valialkin
3230525c36
docs: use persistent links to Grafana dashboards
These links do not depend on the dashboard name, so they do not break after the renaming of the dashboard.

This is a follow-up for ff33e60a3d
2024-01-22 01:45:42 +02:00
Aliaksandr Valialkin
885ee160c2
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-22 01:23:23 +02:00
hagen1778
a99d26633b
docs: remove slug from Grafana dashboard URLs
Each Grafana dashboard has unique ID which can be used to fetch the dashboard
from grafana.com: https://grafana.com/grafana/dashboards/11176
The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/
But using slug implies that any change to dashboard name will break the link.
So it is better to just use ID, so the dashboard URL will never break.

This is follow-up for ff33e60a3d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 00:23:30 +02:00
Aliaksandr Valialkin
6feef14095
lib/protoparser: add missing /datadog/ prefix to the /api/v2/series path in the description for -datadog.maxInsertRequestSize command-line flag 2023-12-21 21:05:24 +02:00
Aliaksandr Valialkin
62a105d9e9
app/{vminsert,vmagent}: preliminary support for /api/v2/series ingestion from new versions of DataDog Agent
This commit adds only JSON support - https://docs.datadoghq.com/api/latest/metrics/#submit-metrics ,
while recent versions of DataDog Agent send data to /api/v2/series in undocumented Protobuf format.
The support for this format will be added later.

Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451
2023-12-21 20:50:27 +02:00
Dmytro Kozlov
7afec3d813
docs: remove default value from the maxConcurrentInserts flag (#5494) 2023-12-20 14:15:20 +02:00
Aliaksandr Valialkin
0a99c819bf
all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page
This may be needed for systems, which require this metadata such as Google Cloud Managed Prometheus.
See https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
2023-12-19 03:26:02 +02:00
Aliaksandr Valialkin
9540d29154
lib/pushmetrics: add -pushmetrics.header and -pushmetrics.disableCompression command-line flags 2023-12-17 19:58:14 +02:00
Aliaksandr Valialkin
42629dead1
app/vmstorage: addd missing -inmemoryDataFlushInterval command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2023-12-14 20:47:49 +02:00
Aliaksandr Valialkin
e4bb2808f1
app/vmselect: add support for vmstorage groups with independent -replicationFactor per group
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmstorage-groups-at-vmselect

Thanks to @zekker6 for the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/718
2023-12-13 00:14:34 +02:00
Aliaksandr Valialkin
de1b394af8
docs: sync -help output for VictoriaMetrics components after recent changes 2023-12-10 01:13:51 +02:00
Aliaksandr Valialkin
3d6517b05e
app/vmselect: add -search.maxResponseSeries command-line flag for limiting the number of time series a single response can return
This limit can be used for preventing from high memory usage at Grafana when the response returns too many series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5372
2023-12-10 00:54:32 +02:00
Andrii Chubatiuk
4ed7da8c58
docs: sync mistakenly deleted docs from 543f218fe9
543f218fe9
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 48228031e4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-12-01 13:59:51 +01:00
Ivan Yatskevich
bfaca07774
docs/dns-srv-typo-fix: replace dns+src with dns+srv (#5396) 2023-11-28 17:46:00 +02:00
Aliaksandr Valialkin
f2346a79b6
docs/Cluster-VictoriaMetrics.md: document that multitenancy via labels is applied to data ingested via non-http protocols
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3009
2023-11-27 11:18:15 +02:00
Aliaksandr Valialkin
3674232128
docs: make more visible that the maximum JSON line length, which is accepted by /api/v1/import, is limited by -import.maxLineLen command-line flag value
This is a follow-up for 0cf55ded34

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5364
2023-11-24 13:14:40 +02:00
Aliaksandr Valialkin
db6dadf1f7
docs: convert png images to webp in all the docs except of docs/operator/*
This reduces the size of docs/* folder from 33MB to 18MB

Images inside docs/operator/* must be converted at the https://github.com/VictoriaMetrics/operator/tree/master/docs
and then the updated images must be automatically propagated to the docs/operator/*

This is a follow-up for d3f919df3e

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5206
2023-11-22 19:29:47 +02:00
hagen1778
34b7783461
docs: calrify version when vminsertConnsShutdownDuration was added
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-20 17:14:57 +01:00
Aliaksandr Valialkin
d8220066dc
docs/Cluster-VictoriaMetrics.md: clarify how -storage.vminsertConnsShutdownDuration command-line flag works 2023-11-14 21:41:19 +01:00
Zakhar Bessarab
f7834767c1
vmcluster: re-routing enhancement (#5293)
* app/vmstorage: close vminsert connections gradually before stopping storage

Implements graceful shutdown approach suggested here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1768146878

Test results for this can be found here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1790640274

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update graceful shutdown logic

- close connections from vminsert in determenistic order
- update flag description
- lower default timeout to 25 seconds. 25 seconds value was chosen because the lowest default value used in default configuration deployments is 30s(default value in Kubernetes and ansible-playbooks).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/cluster: add information about re-routing enhancement during restart

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: add entry for new command-line flag

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* {app/vmstorage,lib/ingestserver}: address review feedback

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/cluster: add note to update workload scheduler timeout

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-11-14 01:00:42 +01:00
Aliaksandr Valialkin
d9ecc3f6d7
lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args 2023-11-13 09:43:49 +01:00
Aliaksandr Valialkin
ed79f9806a
lib/blockcache: do not cache entries, which were attempted to be accessed 1 or 2 times
Previously entries which were accessed only 1 time weren't cached.
It has been appeared that some rarely executed heavy queries may read indexdb block twice
in a row instead of once. There is no need in caching such a block then.
This change should eliminate cache size spikes for indexdb/dataBlocks when such heavy queries are executed.

Expose -blockcache.missesBeforeCaching command-line flag, which can be used for fine-tuning
the number of cache misses needed before storing the block in the caching.
2023-11-13 09:38:57 +01:00
Aliaksandr Valialkin
3d6f4da3b3
docs: update -help output after recent changes to VictoriaMetrics components 2023-11-02 20:27:16 +01:00
Aliaksandr Valialkin
bf01a97f17
docs/CHANGELOG.md: update the description of the optimization for SLO/SLI-like queries according to latest changes
See commits 4497a08e3d and 92826b0b4a
2023-11-02 20:09:22 +01:00
Aliaksandr Valialkin
a7a73b9845
docs/Cluster-VictoriaMetrics.md: clarify the description on why -dosnwampling.period must be set at both vmstorage and vmselect
This is a follow-up for ca7457d906
2023-10-31 19:08:21 +01:00
Roman Khavronenko
9d8f93050c
app/vmselect: expose vm_memory_intensive_queries_total counter metric (#5208)
The new metric gets increased each time `-search.logQueryMemoryUsage` memory limit
is exceeded by a query. This metric should help to identify expensive and heavy queries
without inspecting the logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-31 19:02:22 +01:00
Roman Khavronenko
c6d74108fb
docs: explain motivation behind having -downsampling.period on vmselect (#5205)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 15:29:35 +01:00
Dima Lazerka
ed8fc04898
lib/httpserver: add flags to specify HSTS / Frame-Options / CSP headers for httpserver (#5111)
support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options`
HTTP headers in all VictoriaMetrics components.
The values for headers can be specified by users via the following flags:
`-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`.

Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit ad839aa492)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 11:41:38 +01:00
Aliaksandr Valialkin
94e061087f
docs: use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest instead of https://github.com/VictoriaMetrics/VictoriaMetrics/releases link where needed
The https://github.com/VictoriaMetrics/VictoriaMetrics/releases link may show non-latest
releases at the top, such as LTS releases or VictoriaLogs releases.
So it is better to use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest link,
which always redirect to the latest available release of VictoriaMetrics.
2023-10-26 09:23:17 +02:00
Roman Khavronenko
cd2247b24a
app/vmselect: limit the number of parallel workers by 32 (#5195)
* app/vmselect: limit the number of parallel workers by 32

The change should improve performance and memory usage during query processing
on machines with big number of CPU cores. The number of parallel workers for
query processing is controlled via `-search.maxWorkersPerQuery` command-line flag.
By default, the number of workers is limited by the number of available CPU cores,
but not more than 32. The limit can be increased via `-search.maxWorkersPerQuery`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

- The `-search.maxWorkersPerQuery` command-line flag doesn't limit resource usage,
  so move it from the `resource usage limits` to `troubleshooting` chapter at docs/Single-server-VictoriaMetrics.md

- Make more clear the description for the `-search.maxWorkersPerQuery` command-line flag

- Add the description of `-search.maxWorkersPerQuery` to docs/Cluster-VictoriaMetrics.md

- Limit the maximum value, which can be passed to `-search.maxWorkersPerQuery`, to GOMAXPROCS,
  because bigger values may worsen query performance and increase CPU usage

- Improve the the description of the change at docs/CHANGELOG.md. Mark it as FEATURE instead of BUGFIX,
  since it is closer to a feature than to a bugfix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-26 09:15:27 +02:00
Aliaksandr Valialkin
815e9bf892
app/{vmagent,vminsert}: follow-up for NewRelic data ingestion protocol support
This is a follow-up for f60c08a7bd

Changes:

- Make sure all the urls related to NewRelic protocol start from /newrelic . Previously some urls were started from /api/v1/newrelic

- Remove /api/v1 part from NewRelic urls, since it has no sense

- Remove automatic transformation from CamelCase to snake_case for NewRelic labels and metric names,
  since it may complicate the transition from NewRelic to VictoriaMetrics. Preserve all the metric names and label names,
  so users could query metrics and labels by the same names which are used in NewRelic.
  The automatic transformation from CamelCase to snake_case can be added later as a special action for relabeling rules if needed.

- Properly update per-tenant data ingestion stats at app/vmagent/newrelic/request_handler.go . Previously it was always zero.

- Fix NewRelic urls in vmagent when multitenant data ingestion is enabled. Previously they were mistakenly started from `/`.

- Document NewRelic data ingestion url at docs/Cluster-VictoriaMetrics.md

- Remove superflouos memory allocations at lib/protoparser/newrelic

- Improve tests at lib/protoparser/newrelic/*

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712
2023-10-16 13:55:04 +02:00