Commit Graph

5632 Commits

Author SHA1 Message Date
Yury Molodov
9732790935
vmui: minor fixes (#3276)
* feat: apply serverURL on down Enter

* fix: change method of set time range

* fix: remove prevent run fetch without changes

* fix: prevent reset timerange when autorefresh
2022-10-28 14:52:15 +03:00
Aliaksandr Valialkin
ed1056d3c9
docs/Cluster-VictoriaMetrics.md: improve docs for dns+srv service discovery 2022-10-28 14:23:02 +03:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
976bbe3677
app/{vminsert,vmselect}: limit the access to storageNodes to getStorageNodesBucket and setStorageNodesBucket functions
This makes the code more maintainable and earier to test.
2022-10-28 11:41:55 +03:00
Aliaksandr Valialkin
1386b6bd69
vendor: make vendor-update 2022-10-28 00:19:05 +03:00
Aliaksandr Valialkin
7ae038766c
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
8a6898b625
Revert "vmalert: escape query params if external alert source defined (#3267)"
This reverts commit 00c838353d.

Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 .
Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana:

https://grafana.example.com/explore%3ForgId%3D1%26left%3D...

The next commit will contain the correct fix of the issue - the `quotesEscape` function must
properly escape the string, so it could be embedded into JSON string. This function must
properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary.
Actually, the next commit makes the `crlfEscape` function deprecated.
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
f4e9393076
vendor: make vendor-update 2022-10-28 00:08:50 +03:00
Denys Holius
6e5308ecb2
.github/workflows/codeql-analysis.yml: specifically setting the Go version (#3277)
see https://github.com/github/codeql-action/issues/1059
2022-10-28 00:08:50 +03:00
Dmytro Kozlov
3123059407
vmalert: escape query params if external alert source defined (#3267)
vmalert: escape query args if external alert source defined
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin
46450995ae
app/vmselect/vmui: make vmui-update after eae6063450 2022-10-26 02:51:09 +03:00
Yury Molodov
1847a207f8
fix: change step setting field (#3270)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-26 02:47:41 +03:00
Yury Molodov
346d49ae61
vmui: limit the number of decimal places to 10 characters (#3258)
* fix: limit the number of decimal places to 10 characters

* fix: add float numbers stabilizer
2022-10-26 02:43:43 +03:00
Yury Molodov
ab880afa50
vmui: add responsive styles for small screens (#3256)
* fix: add responsive styles for small screens

* fix: correct additional settings margins

* docs/CHANGELOG.md: add responsive styles
2022-10-26 02:40:41 +03:00
Aliaksandr Valialkin
c92d1fdf8e
docs/Cluster-VictoriaMetrics.md: update docs about env vars usage in command-line flags
This is a follow-up for 02096e06d0
2022-10-26 01:55:25 +03:00
Aliaksandr Valialkin
36b92f07f7
lib/envflag: allow referring environment variables in command-line flags 2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin
75990de795
docs: run make docs-sync after 2ed3d49c26 2022-10-26 01:14:54 +03:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
9d0652fc6e
docs/Makefile: fix docs-up Makefile command after 9ccd22c1f6 2022-10-25 17:54:14 +03:00
Denys Holius
8c0f93aa55
Docs: add guide "How to delete and replace metrics" (#2829)
docs: add guide how to delete and replace metrics
2022-10-25 17:54:13 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
596251eb87
app/vmselect/vmui: make vmui-update after 274e235bf7 2022-10-24 21:29:57 +03:00
Aliaksandr Valialkin
608c977978
docs/guides/migrate-from-influx.md: properly display images at https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/guides/migrate-from-influx.md
This is a follow-up for 42375679db
2022-10-24 21:29:55 +03:00
Nguyen Van Duc
f7a1001b55
Fix images not display on key concepts document (#3266) 2022-10-24 21:23:38 +03:00
Aliaksandr Valialkin
46a37fbeac
docs/CHANGELOG.md: document 274e235bf7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240
2022-10-24 21:03:03 +03:00
Yury Molodov
96a579c13b
vmui: optimize memory (#3255)
* fix: change series limit logic

* fix: remove spread operators
2022-10-24 21:03:02 +03:00
Yury Molodov
5de090ab43
vmui: extend options for app mode (#3252)
* feat: add vmui customization for dbaas

* feat: extends vmui customization for dbaas

* fix: move input tenandId after query

* fix: change serverURL when changing tenantID

* fix: remove options

* docs: add options description
2022-10-24 20:49:46 +03:00
Aliaksandr Valialkin
8ea84432ef
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin
754208b124
docs/CHANGELOG.md: typo fixes 2022-10-24 17:00:18 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
22a5405cfa
docs/CHANGELOG.md: document 69b27275d2b2bf1bdae0d8b887b44bde2a787649 2022-10-24 16:08:35 +03:00
Aliaksandr Valialkin
8946371466
vendor: make vendor-update 2022-10-24 15:50:57 +03:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
2fc82b846e
lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg
This makes code easier to read.

This is a follow-up after d2d30581a0
2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin
d51f9b9284
lib/storage: small code cleanups 2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin
5ace1587e6
lib/storage: re-use newTestStorage() instead of manually initializing Storage mock
This is a follow-up for d2d30581a0
2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin
57ea7a3ee8
lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback
This improves code readability a bit.
2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin
63419d8e7c
lib/storage: small refactoring: move retentionDeadline to blockStreamMerger
This allows defining per-block retention in the future by updating the getRetentionDeadline function
2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin
31071347ca
lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop 2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin
5d0a91afd5
lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms 2022-10-23 12:48:43 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
c92aef39b5
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:14:07 +03:00
Aliaksandr Valialkin
fe5611d6e1
lib/storage: free up memory occupied by Storage.pendingHourEntries after a temporary spike in its memory usage
This reduces vmstorage memory usage by up to 20% in production workload
2022-10-21 14:59:14 +03:00
Aliaksandr Valialkin
32b6ce691b
lib/storage: move common code to newRawRowsBlock() function 2022-10-21 14:46:06 +03:00
Aliaksandr Valialkin
2f8861ed9c
lib/storage: simplify code a bit after 3f5959c053 2022-10-21 14:39:44 +03:00
Aliaksandr Valialkin
1fb2be0cae
lib/{mergeset,storage}: simplify the code a bit after ae55ad8749 2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
This reduces CPU usage when there is no sense in validating timestamps.

This is a follow-up for 5fa9525498

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
edf3b7be47
lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate
This should improve background merge rate under high load a bit
2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin
4d71023eb9
lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes
This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes.
This

This change shouldn't increase memory usage for caches, since the prev cache cleaner
should free up memory by deleting unused prev cache as soon as possible.
See 08ca45d238 for details on prev cache cleaner.
2022-10-20 21:59:08 +03:00