Aliaksandr Valialkin
fd9d0a550b
lib/storage: remove logging redundant path values in a single error message
2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin
82f64072d2
lib/filestream: remove logging redundant path values in a single error message
2022-12-03 22:02:04 -08:00
Aliaksandr Valialkin
81400c80f0
lib/fs: remove logging redundant path values in a single error message
2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin
dc890ab80c
lib/backup: remove logging duplicate path values in a single error message
2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin
6910b1de2e
all: typo fix: the the -> the
2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin
0b8e7deabd
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
...
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin
8e9822bc7f
lib/storage: speed up search for data block for the given tsids
...
Use binary search instead of linear scan for looking up the needed
data block inside index block.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425
2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin
d8303845ef
lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC
2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin
d8d4d21d7a
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts
...
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin
be6da5053f
lib/promscrape: optimize service discovery speed
...
- Return meta-labels for the discovered targets via promutils.Labels
instead of map[string]string. This improves the speed of generating
meta-labels for discovered targets by up to 5x.
- Remove memory allocations in hot paths during ScrapeWork generation.
The ScrapeWork contains scrape settings for a single discovered target.
This improves the service discovery speed by up to 2x.
2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin
2524d94fe1
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels
2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin
027ab74efb
lib/httpserver: link to url format docs in error message emitted on parse error for the provided url path
...
This should help users identifying and fixing improperly set up urls.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402
2022-11-28 18:49:03 -08:00
Aliaksandr Valialkin
8ce5b095b7
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
2022-11-28 18:37:34 -08:00
匠心零度
d4808d5b84
lib/storage: remove extra error check ( #3396 )
2022-11-28 17:07:11 +01:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
...
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:42:38 -08:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants ( #3348 )
...
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* app/vmselect: fix error message
* {app/vmstorage,app/vmselect}: fix error messages
* app/vmselect: change log level for error handling
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 10:32:45 -08:00
Roman Khavronenko
d1169c1559
vmagent: expose metrics for tracking config state ( #3375 )
...
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin
c33bcae457
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones
...
The filter arg isn't supported by zones API in GCE.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin
48dda17e36
lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration
...
While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-17 21:55:11 +02:00
Aliaksandr Valialkin
047fe3ee67
lib/promscrape: add a benchmark for internLabelStrings()
2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin
b0b8f05fa4
lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS()
...
The issue has been introduced in 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin
c8d7e1312c
lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio
...
Reduce the default value of this flag from 1% to 0.2% after 71335e6024https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:41:37 +02:00
Aliaksandr Valialkin
06300fe9b8
lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf
...
Previously indexBlock.bhs pointed to the buffer, which could be changed over time.
This could result in incorrect time series search over time.
This is a follow-up for 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin
454060fd78
lib/mergeset: remove string allocation and copying when unmarshaling blockHeader
...
This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin
a5a0fa75bf
lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1%
...
It has been appeared that some production workloads could suffer for some time
after every reset of the previous cache when it gets less than 5% of requests
after the needed item isn't found in the current cache. This could result
in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM
usage needed for reading, unpacking and caching the missed data from disk.
This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%.
This should reduce the possible negative impact after each cache reset by at least 5x,
while reducing the total memory used by caches.
This is a follow-up for d906d8573e
2022-11-10 12:38:53 +02:00
Aliaksandr Valialkin
24213eaeba
lib/promscrape: add more cases to TestAddRowToTimeseries
...
This is a follow-up for 16fdd2af8a
2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL
87375b004a
test(golang): add test to function addRowToTimeseries ( #3282 )
...
Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>
2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin
2091693f16
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
...
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:36:15 +02:00
Roman Khavronenko
71dfe4d697
protoparser/opentsdb: allow lines without tags ( #3303 )
...
According to http://opentsdb.net/docs/build/html/api_telnet/put.html
"At least one tag pair must be present".
However, in VictoriaMetrics datamodel tags aren't required.
This could be confusing for users. Allowing accept lines without
tags seems to do no harm.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin
abf7e4e72f
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does
...
See https://github.com/prometheus/prometheus/pull/11482
2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin
d3035b1ca1
lib/storage: follow-up for 790768f20b
...
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin
be78950011
lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8
2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin
9d901ee55a
lib/envtemplate: allow non-env var names inside "%{ ... }"
2022-11-07 13:16:00 +02:00
Aliaksandr Valialkin
99e6a937a5
lib/storage: remove unused isFull field from hourMetricIDs struct
2022-11-07 13:15:59 +02:00
Aliaksandr Valialkin
4a6d5ab1b1
lib/promrelabel: go fmt after 5cec9706dc
2022-10-29 05:17:49 +03:00
Aliaksandr Valialkin
a72bf87e04
lib/promrelabel: add a test from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251
2022-10-29 04:34:08 +03:00
Aliaksandr Valialkin
eae334f70e
lib/envflag: small refactoring after 518c340ae3 and 02096e06d0
2022-10-29 02:29:19 +03:00
Aliaksandr Valialkin
ac5528cb46
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set.
...
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.
While at it, properly handle some corner cases:
- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278
Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin
ba739c8052
lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page
2022-10-28 22:15:30 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
...
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin
36b92f07f7
lib/envflag: allow referring environment variables in command-line flags
2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists
2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
...
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.
This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search
...
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
2fc82b846e
lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg
...
This makes code easier to read.
This is a follow-up after d2d30581a0
2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin
d51f9b9284
lib/storage: small code cleanups
2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin
5ace1587e6
lib/storage: re-use newTestStorage() instead of manually initializing Storage mock
...
This is a follow-up for d2d30581a0
2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin
57ea7a3ee8
lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback
...
This improves code readability a bit.
2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin
63419d8e7c
lib/storage: small refactoring: move retentionDeadline to blockStreamMerger
...
This allows defining per-block retention in the future by updating the getRetentionDeadline function
2022-10-23 16:11:01 +03:00
