Commit Graph

218 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
c06ec450ee
docs/Cluster-VictoriaMetrics.md: move Naive_cluster_scheme.png from the docs/assets/images/ folder into docs/ folder and add Cluster-VictoriaMetrics_ prefix to the image name
The docs/assets folder should be used only for assets specific to docs generation at https://docs.victoriametrics.com, e.g. css, js and images.

All the other assets related to specific docs should be placed in the same folder as the corresponding *.md file.
These assets should have the same name prefix as the corresponding doc file name. This simplifies tracking the lifetime of these assets.
For example, if the doc is removed, it is very easy to remove all assets associated with it with a simple `rm -rf docs/doc-name*` command.

This also simplifies generating correct urls for doc-specific assets from both https://docs.victoriametrics.com
and from https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/ - just refer to the asset name without any directory prefixes.
2023-01-27 11:31:21 -08:00
Aliaksandr Valialkin
9e8cfd23ad
docs/Cluster-VictoriaMetrics.md: update command-line descriptions after ebebaecd94 2023-01-27 00:04:54 -08:00
Aliaksandr Valialkin
a6988eb8c3
docs/Cluster-VictoriaMetrics.md: mention the -vmui.customDashboardsPath command-line flag at vmselect
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322
2023-01-11 23:39:18 -08:00
Aliaksandr Valialkin
5876821a16
all: small improvements in error messages and command-line flag descriptions related to concurrency limiters 2023-01-07 00:12:24 -08:00
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin
20e9598254
lib/vmselectapi: limit the number of concurrently executed requests
This should prevent from out of memory errors when big number of vmselect
nodes send many concurrent requests to vmstorage

The limit can be controlled at vmstorage via the following command-line flags:
- search.maxConcurrentRequests
- search.maxQueueDuration

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits
2023-01-06 18:39:46 -08:00
Aliaksandr Valialkin
ac890b3081
docs: update -help outputs for vm* tools 2023-01-03 23:27:31 -08:00
Artem Navoiev
1ae98867a5
fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-28 11:30:53 -08:00
Artem Navoiev
393f4ab86f
update links to grafana dashboards (#3534)
docs: update links to grafana dashboards

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-28 11:22:02 -08:00
Roman Khavronenko
f8a3514e99
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 17:14:23 -08:00
Aliaksandr Valialkin
3a28a52667
lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes 2022-12-14 17:53:18 -08:00
Aliaksandr Valialkin
7d1d82d094
docs/Cluster-VictoriaMetrics.md: mention the vm_storage_is_read_only metric, which can help debugging readonly mode at vmstorage 2022-12-14 09:32:21 -08:00
Aliaksandr Valialkin
97b41e727c
lib/promscrape: implement target-level and metric-level relabel debugging
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
2022-12-10 02:25:56 -08:00
Aliaksandr Valialkin
81da751e27
docs/Cluster-VictoriaMetrics.md: typo fix 2022-12-07 12:26:45 -08:00
Aliaksandr Valialkin
d009ba1def
docs/Cluster-VictoriaMetrics.md: make docs-sync after 5de8330ce00adfc5ac794070d30a2617ddc14bf2 2022-12-07 12:02:28 -08:00
Aliaksandr Valialkin
6680a17fec
docs: document the addition of -storageNode.discoveryInterval command-line flag 2022-12-06 19:57:46 -08:00
Roman Khavronenko
2d8ac5355b
docs: fix typo in cluster's README (#3430)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-02 19:11:45 -08:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:42:38 -08:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348)
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* app/vmselect: fix error message

* {app/vmstorage,app/vmselect}: fix error messages

* app/vmselect: change log level for error handling

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 10:32:45 -08:00
Aliaksandr Valialkin
6fe8eec745
all: add a link to https://docs.victoriametrics.com/enterprise.html into description for enterprise flags 2022-11-21 15:44:54 +02:00
Aliaksandr Valialkin
0e51967970
app/{vminsert,vmselect}: add -storageNode.filter command-line flag for filtering the discovered storage nodes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353
2022-11-21 15:20:20 +02:00
Aliaksandr Valialkin
97eafbe4a7
app/vmselect: clarify that it isnt recommended setting -replicationFactor at vmselect nodes even if the replication is enabled at vminsert nodes 2022-11-18 14:04:12 +02:00
Aliaksandr Valialkin
c8d7e1312c
lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio
Reduce the default value of this flag from 1% to 0.2% after 71335e6024

This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:41:37 +02:00
Zakhar Bessarab
fd36c9b3b1
docs/{vminsert,vmselect}: clarify flag description for clusternativeListenAddr (#3314)
docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`
2022-11-04 09:49:53 +01:00
Aliaksandr Valialkin
ac7c43c246
docs: typo fixes 2022-10-29 04:53:03 +03:00
Aliaksandr Valialkin
ed1056d3c9
docs/Cluster-VictoriaMetrics.md: improve docs for dns+srv service discovery 2022-10-28 14:23:02 +03:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin
c92d1fdf8e
docs/Cluster-VictoriaMetrics.md: update docs about env vars usage in command-line flags
This is a follow-up for 02096e06d0
2022-10-26 01:55:25 +03:00
Aliaksandr Valialkin
8ea84432ef
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
098c9bda27
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin
d8057fca0b
docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag
This is a follow-up for 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-09 13:58:18 +03:00
Aliaksandr Valialkin
938ff7bba6
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-08 01:14:01 +03:00
Aliaksandr Valialkin
e6ee725bae
docs/Cluster-VictoriaMetrics.md: add a security note for multitenant support via labels 2022-10-01 18:56:56 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
nemobis
6723bdf867
docs: fix typo in Cluster-VictoriaMetrics.md (#3172) 2022-09-28 10:43:17 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin
6a6cf9c590
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:13:31 +03:00
Roman Khavronenko
c027991caf
docs: mention VictoriaMetrics Monitoring blog post (#3152)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:29:02 +03:00
Roman Khavronenko
f772ee8326
deployment/docker: move cluster compose env to master branch (#3130)
* deployment/docker: move cluster compose env to master branch

The change supposed to simplify the process of maintaining for
single/cluster docker-compose envs, alerts, dashboards. It also
supposes to reduce confusion for users when looking for cluster
related alerts/configs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/docker: move cluster compose env to master branch

Review updates.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-21 12:03:10 +03:00
Aliaksandr Valialkin
27a6ac5a30
docs/Cluster-VictoriaMetrics.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:19:11 +03:00
Aliaksandr Valialkin
74efe00649
docs: clarify why cluster version is recommended to use for serving high loads 2022-08-30 13:21:43 +03:00
Dmytro Kozlov
d32a6359b0
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:27:41 +03:00
Aliaksandr Valialkin
e2d8916935
docs: mention that it is safe sharing the collected profiles from security PoV
The collected profiles do not contain sensitive information
2022-08-24 14:08:30 +03:00
Aliaksandr Valialkin
7b9ba456ff
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:

    vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:41:57 +03:00
Ivan Dudin
c7ca124803
Update Cluster-VictoriaMetrics.md (#3005)
Flags `-search.maxTagKeys` and `-search.maxTagValues` are present at vmstorage and not at vmselect
```
  -search.maxTagKeys int
  -search.maxTagValues int
```
2022-08-21 23:30:15 +03:00
Aliaksandr Valialkin
7f46c17e4e
docs/Cluster-VictoriaMetrics.md: document the best strategies for cluster update / upgrade 2022-08-21 23:20:26 +03:00