Aliaksandr Valialkin
24213eaeba
lib/promscrape: add more cases to TestAddRowToTimeseries
...
This is a follow-up for 16fdd2af8a
2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL
87375b004a
test(golang): add test to function addRowToTimeseries ( #3282 )
...
Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>
2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin
2091693f16
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
...
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:36:15 +02:00
Roman Khavronenko
71dfe4d697
protoparser/opentsdb: allow lines without tags ( #3303 )
...
According to http://opentsdb.net/docs/build/html/api_telnet/put.html
"At least one tag pair must be present".
However, in VictoriaMetrics datamodel tags aren't required.
This could be confusing for users. Allowing accept lines without
tags seems to do no harm.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin
abf7e4e72f
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does
...
See https://github.com/prometheus/prometheus/pull/11482
2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin
d3035b1ca1
lib/storage: follow-up for 790768f20b
...
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin
be78950011
lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8
2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin
9d901ee55a
lib/envtemplate: allow non-env var names inside "%{ ... }"
2022-11-07 13:16:00 +02:00
Aliaksandr Valialkin
99e6a937a5
lib/storage: remove unused isFull field from hourMetricIDs struct
2022-11-07 13:15:59 +02:00
Aliaksandr Valialkin
4a6d5ab1b1
lib/promrelabel: go fmt after 5cec9706dc
2022-10-29 05:17:49 +03:00
Aliaksandr Valialkin
a72bf87e04
lib/promrelabel: add a test from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251
2022-10-29 04:34:08 +03:00
Aliaksandr Valialkin
eae334f70e
lib/envflag: small refactoring after 518c340ae3 and 02096e06d0
2022-10-29 02:29:19 +03:00
Aliaksandr Valialkin
ac5528cb46
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set.
...
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.
While at it, properly handle some corner cases:
- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278
Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin
ba739c8052
lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page
2022-10-28 22:15:30 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
...
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin
36b92f07f7
lib/envflag: allow referring environment variables in command-line flags
2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists
2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
...
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.
This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search
...
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
2fc82b846e
lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg
...
This makes code easier to read.
This is a follow-up after d2d30581a0
2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin
d51f9b9284
lib/storage: small code cleanups
2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin
5ace1587e6
lib/storage: re-use newTestStorage() instead of manually initializing Storage mock
...
This is a follow-up for d2d30581a0
2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin
57ea7a3ee8
lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback
...
This improves code readability a bit.
2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin
63419d8e7c
lib/storage: small refactoring: move retentionDeadline to blockStreamMerger
...
This allows defining per-block retention in the future by updating the getRetentionDeadline function
2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin
31071347ca
lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop
2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin
5d0a91afd5
lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms
2022-10-23 12:48:43 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
...
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.
The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.
This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.
This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
fe5611d6e1
lib/storage: free up memory occupied by Storage.pendingHourEntries after a temporary spike in its memory usage
...
This reduces vmstorage memory usage by up to 20% in production workload
2022-10-21 14:59:14 +03:00
Aliaksandr Valialkin
32b6ce691b
lib/storage: move common code to newRawRowsBlock() function
2022-10-21 14:46:06 +03:00
Aliaksandr Valialkin
2f8861ed9c
lib/storage: simplify code a bit after 3f5959c053
2022-10-21 14:39:44 +03:00
Aliaksandr Valialkin
1fb2be0cae
lib/{mergeset,storage}: simplify the code a bit after ae55ad8749
2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
...
This reduces CPU usage when there is no sense in validating timestamps.
This is a follow-up for 5fa9525498https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
edf3b7be47
lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate
...
This should improve background merge rate under high load a bit
2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin
4d71023eb9
lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes
...
This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes.
This
This change shouldn't increase memory usage for caches, since the prev cache cleaner
should free up memory by deleting unused prev cache as soon as possible.
See 08ca45d238
2022-10-20 21:59:08 +03:00
Aliaksandr Valialkin
9a52b56b89
lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine
...
This makes the code more clear after d906d8573e
2022-10-20 21:59:02 +03:00
Aliaksandr Valialkin
324e119172
lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal
...
Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals
if they hang for some reason in graceful shutdown procedure.
2022-10-20 21:58:49 +03:00
Aliaksandr Valialkin
6855de311c
lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms
...
The panic has been introduced in 68f3a02589
2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin
526bc8a8b0
lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache
...
This means that the majority of requests are successfully served from the current cache,
so the previous cache can be reset in order to free up memory.
2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin
42cda38dbc
lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses
...
This should improve cache scalability on systems with many CPU cores.
2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin
f22bea242f
lib/workingsetcache: randomize interval for swapping curr and prev caches
...
This should make CPU usage smoother over time, since different caches
will be swapped at different times.
2022-10-20 10:48:46 +03:00
Nikolay
ea0596d9d8
lib/promscrape/discovery/kubernetes: correctly wrap error ( #3250 )
...
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201
2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin
d0288ea417
all: log error when environment variables referred from -promscrape.config are missing
...
This should prevent from using incorrect config files
2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin
e4e2d1fcde
lib/protoparser/clusternative: allocate unmarshalWork after reading the data from input connection
...
This shortens the time when unmarshalWork is in use. This also reduces the number of unmarshalWork
objects in the pool, and its memory usage.
2022-10-18 00:24:04 +03:00
Aliaksandr Valialkin
481ca746ba
lib/protoparser/clusternative: reuse unmarshalWork in order to reduce memory allocations
2022-10-18 00:06:56 +03:00
Aliaksandr Valialkin
6f69a88a5a
lib/storage: double the number of rawRows shards on multi-core systems
...
This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage
2022-10-17 18:19:28 +03:00
Aliaksandr Valialkin
68f3a02589
lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list
2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin
c4a3d8b169
lib/promrelabel: add relabeling tests when the source label is missing
2022-10-17 14:48:29 +03:00
Aliaksandr Valialkin
ed324aad66
lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map
...
This prevents from possible corruption of the internal cache map
when the underlying byte slice used by the string key is modified.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227
2022-10-14 09:52:18 +03:00
Nikolay
07140e0877
lib/backup: set s3 default region to us-west-2 ( #3224 )
...
* lib/backup: set s3 default region to us-west-2
it should fix an error with region detection for bucket, if AWS_REGION env var is not set
* Update lib/backup/s3remote/s3.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:06:24 +03:00
