Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
...
This reduces CPU usage when there is no sense in validating timestamps.
This is a follow-up for 5fa9525498https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
edf3b7be47
lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate
...
This should improve background merge rate under high load a bit
2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin
4d71023eb9
lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes
...
This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes.
This
This change shouldn't increase memory usage for caches, since the prev cache cleaner
should free up memory by deleting unused prev cache as soon as possible.
See 08ca45d238
2022-10-20 21:59:08 +03:00
Aliaksandr Valialkin
9a52b56b89
lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine
...
This makes the code more clear after d906d8573e
2022-10-20 21:59:02 +03:00
Aliaksandr Valialkin
324e119172
lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal
...
Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals
if they hang for some reason in graceful shutdown procedure.
2022-10-20 21:58:49 +03:00
Aliaksandr Valialkin
6855de311c
lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms
...
The panic has been introduced in 68f3a02589
2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin
526bc8a8b0
lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache
...
This means that the majority of requests are successfully served from the current cache,
so the previous cache can be reset in order to free up memory.
2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin
42cda38dbc
lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses
...
This should improve cache scalability on systems with many CPU cores.
2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin
f22bea242f
lib/workingsetcache: randomize interval for swapping curr and prev caches
...
This should make CPU usage smoother over time, since different caches
will be swapped at different times.
2022-10-20 10:48:46 +03:00
Nikolay
ea0596d9d8
lib/promscrape/discovery/kubernetes: correctly wrap error ( #3250 )
...
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201
2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin
d0288ea417
all: log error when environment variables referred from -promscrape.config are missing
...
This should prevent from using incorrect config files
2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin
e4e2d1fcde
lib/protoparser/clusternative: allocate unmarshalWork after reading the data from input connection
...
This shortens the time when unmarshalWork is in use. This also reduces the number of unmarshalWork
objects in the pool, and its memory usage.
2022-10-18 00:24:04 +03:00
Aliaksandr Valialkin
481ca746ba
lib/protoparser/clusternative: reuse unmarshalWork in order to reduce memory allocations
2022-10-18 00:06:56 +03:00
Aliaksandr Valialkin
6f69a88a5a
lib/storage: double the number of rawRows shards on multi-core systems
...
This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage
2022-10-17 18:19:28 +03:00
Aliaksandr Valialkin
68f3a02589
lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list
2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin
c4a3d8b169
lib/promrelabel: add relabeling tests when the source label is missing
2022-10-17 14:48:29 +03:00
Aliaksandr Valialkin
ed324aad66
lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map
...
This prevents from possible corruption of the internal cache map
when the underlying byte slice used by the string key is modified.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227
2022-10-14 09:52:18 +03:00
Nikolay
07140e0877
lib/backup: set s3 default region to us-west-2 ( #3224 )
...
* lib/backup: set s3 default region to us-west-2
it should fix an error with region detection for bucket, if AWS_REGION env var is not set
* Update lib/backup/s3remote/s3.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
7a6e5f9224
lib/mergeset: mention in the error message the path to the part, which triggered the error
...
This should improve debuggability
2022-10-12 09:54:42 +03:00
Aliaksandr Valialkin
087393bcef
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places
...
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.
This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin
3b828535f0
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis
...
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin
3987b0abd1
lib/promscrape: allow specifying full target url in __address__ label
...
Previously the `__address__` label could contain only `host:port` part of the target url,
while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__`
labels. Now it is possible to set the full url in `__address__` label.
This makes valid the following scrape config, which is frequently used by novice users:
scrape_configs:
- job_name: foo
static_configs:
- targets:
- http://host1/metrics1
- https://host2/metrics2
2022-10-07 22:46:29 +03:00
Aliaksandr Valialkin
f926db1de4
lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511
2022-10-07 01:04:37 +03:00
Zakhar Bessarab
a5861407cc
app/vmbackup: fix compatibility with latest azure sdk ( #461 )
2022-10-07 01:04:37 +03:00
Aliaksandr Valialkin
958c1f291c
app: follow-up after ec04fcac93
...
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:54:15 +03:00
Dmytro Kozlov
4064db27a8
Properly parse json when export import metric ( #3180 )
...
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API
* app/vmselect: update convert function
* app/vmselect: export null if `math.IsNaN(v)`
* app/vmselect: get float from json
* lib/protoparser: add test
* docs: add change log
* lib/protoparser: make export import api compatible
2022-10-06 14:54:14 +03:00
Zakhar Bessarab
db791a254b
lib/backup/s3remote: fix error checking for alternative S3 providers ( #3191 )
2022-10-06 13:37:23 +03:00
Aliaksandr Valialkin
cc0d70c3d6
lib/backup/azremote: remove unused methods after the 262ce77e2d
2022-10-06 13:30:47 +03:00
Zakhar Bessarab
6a6dcc059b
lib/backup: add support of Azure Blob Storage ( #460 )
...
* lib/backup: add support of Azure Blob Storage
* lib/backup: add enterprise support of Azure Blob Storage
2022-10-06 00:36:19 +03:00
Aliaksandr Valialkin
b857365b84
app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data
...
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin
6f9ce3f6d6
lib/flagutil: rename Array to ArrayString
...
This makes the ArrayString more consistent with other Array* types.
While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
2022-10-01 18:28:19 +03:00
Zakhar Bessarab
5b7e8d1309
vmbackup: update AWS SDK to v2 ( #3174 )
...
* lib/backup/s3remote: update AWS SDK to v2
* Update lib/backup/s3remote/s3.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
* lib/backup/s3remote: refactor error handling
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-01 17:13:04 +03:00
Aliaksandr Valialkin
93e84a1c57
lib/httpserver: use 302 redirects instead of 301 redirects
...
Incorrect 301 redirects can be cached by user agents such as web browsers.
This can complicate recovery procedure after the incorrect redirect is fixed,
e.g. web browser cache must be reset.
The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:56:43 +03:00
Aliaksandr Valialkin
f0a748a3aa
lib/promscrape/discovery/azure: remove unneeded conversion to string
2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin
735de9ee54
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics
...
This aligns with Prometheus behaviour.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin
e5aa34b2e3
lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests
2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin
b96fe2e265
lib/storage: optimize matching speed for non-trivial regexp filters
...
Wrap re.Match into bytesutil.FastStringMatcher.
This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
969ae90941
lib/promrelabel: remove redundant memory allocations by using interned strings
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
d8d455856c
lib/promrelabel: add a benchmark for realistic Kubernetes relabeling
...
The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes
This benchmark has been copied from d521933053/model/relabel/relabel_test.go (L505)https://github.com/prometheus/prometheus/pull/11147
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
c628f5b6eb
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
...
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:49:08 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels ( #3009 )
...
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
* Update docs/Cluster-VictoriaMetrics.md
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* wip
* app/vminsert/netstorage: clean remaining labels in order to free up GC
* docs/Cluster-VictoriaMetrics.md: typo fix
* wip
* wip
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin
39ba55dbb3
lib/promrelabel: go fmt
2022-09-30 12:28:27 +03:00
Aliaksandr Valialkin
9fc2817f41
lib/promrelabel: optimize action: replace for non-trivial regex values
...
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.
Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
f38c9db74d
lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481
2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
fa46c28c5f
lib/promrelabel: optimize action: labelmap for non-trivial regexs
2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
b4bb1477fe
lib/regexutil: cache MatchString results for unoptimized regexps
...
This increases relabeling performance by 3x for unoptimized regexs
2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
f1eebc0a99
lib/promrelabel: properly parse regex with escaped $ at the end
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131
Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:20:57 +03:00
Nikolay
909709346e
lib/awsapi: fixes sign encoding ( #3183 )
...
* lib/awsapi: fixes sign encoding
previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171
* Update lib/awsapi/sign.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:49:18 +03:00
Aliaksandr Valialkin
c0aa10bd73
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil
...
lib/bytesutil is more appropriate place for InternString() function
2022-09-30 07:34:59 +03:00
Aliaksandr Valialkin
4afa25fb38
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed
2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
9c6c691471
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
...
This is a follow-up for 7c2474dac7https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
7f0b95b50a
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
...
Optimize this function by using results cache for input strings.
Use this function all over the code.
This is a follow-up for fcffdba9dchttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin
41882222d3
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
...
This should simplify further code maintenance and refactoring
This is a follow-up after 6ab1cede62
2022-09-26 17:59:07 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server ( #3109 )
...
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server
* lib/httpserver: added descriptions about tls supported versions
* lib/netutil: check minimal tls version, added supported tls versions to error
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Roman Khavronenko
fe71c73fe1
lib/mergeset: follow-up after a0e7432e42 ( #3145 )
...
* lib/mergeset: follow-up after a0e7432e42
2022-09-26 16:43:17 +03:00
Zakhar Bessarab
6c65ee18d9
vmbackup: configure retries for GCS remote FS ( #3156 )
2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin
2b98f2bc1a
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102
See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
...
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
匠心零度
9777c7a367
lib/querytracer: fix comment ( #3135 )
2022-09-22 13:59:17 +02:00
Aliaksandr Valialkin
d1b9cbcef4
lib/promscrape: typo fix after 74c00a8762
2022-09-14 15:08:42 +03:00
Aliaksandr Valialkin
2351468bc4
lib/promscrape: read response body into memory in stream parsing mode before parsing it
...
This reduces scrape duration for targets returning big responses.
The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
2022-09-14 13:29:39 +03:00
Aliaksandr Valialkin
592612b63f
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error
...
This should improve debugging issues with Kubernetes API server
2022-09-13 19:37:40 +03:00
Aliaksandr Valialkin
5b488a339d
lib/mergeset: atomically remove part dirs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin
fe52378f45
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic
2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin
6c9729d694
lib/storage: atomically remove parts inside partitions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 15:28:41 +03:00
Aliaksandr Valialkin
daa42e4f79
lib/storage: atomically remove partitions, which went outside the configured retention
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin
0a342f04b2
lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there
...
Previously the cache directory was removed. This could result in error when the cache directory
is mounted to a separate filesystem.
2022-09-13 13:32:05 +03:00
Aliaksandr Valialkin
ff7188b6a5
lib/storage: atomically remove snapshot directories
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 13:25:48 +03:00
Aliaksandr Valialkin
051e722112
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:07:49 +03:00
Bryce Lampe
5f7f1d5aea
Support "HTTP" and "HTTPS" schemes ( #3019 )
...
* Support "HTTP" and "HTTPS" schemes
* Update lib/promscrape/config.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2022-08-27 02:23:52 +03:00
Aliaksandr Valialkin
04761419ba
lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache
...
This is a follow-up for c06e7a142c
2022-08-27 00:29:59 +03:00
Aliaksandr Valialkin
86394b4179
lib/promscrape: optimize discoveryutils.SanitizeLabelName()
...
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin
cead9c1e67
lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix
...
This should improve slow path speed for regexps without literal prefixes
2022-08-26 21:48:09 +03:00
Aliaksandr Valialkin
427d69e775
lib/promrelabel: optimize common regex mismatch cases for action: replace and action: labelmap
2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin
da7697fda4
lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop,labelkeep,drop and keep
...
This makes possible optimizing additional cases inside regexutil.PromRegex
2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin
e1bd38fa97
lib/promrelabel: optimize matching for commonly used regex patterns in if option
...
The following regex patterns are optimized:
- literal string match, e.g. "foo"
- prefix match, e.g. "foo.*" and "foo.+"
- substring match, e.g. ".*foo.*" and ".+foo.+"
- alternate values match, e.g. "foo|bar|baz"
2022-08-26 14:55:13 +03:00
Aliaksandr Valialkin
c49751adf8
lib/regexutil: add Simplify() function for simplifying the regular expression
2022-08-26 11:57:43 +03:00
Aliaksandr Valialkin
909e681024
lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep} with anchored regex prefix
...
The following commonly used relabeling rules must work faster now:
- action: labeldrop
regex: "^foo.+$"
- action: labeldrop
regex: "^bar.*"
2022-08-25 23:24:38 +03:00
Aliaksandr Valialkin
d60654eb0a
lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values
...
For example, the following relabeling rule must work much faster now:
- action: labeldrop
regex: "foo|bar|baz"
2022-08-24 17:55:54 +03:00
Aliaksandr Valialkin
891eb608df
lib/storage: increase the maximum possible or values extracted from regexp from 20 to 100
...
This should improve time series search speed for regexp filters with big number of `or` values.
2022-08-24 17:16:29 +03:00
Aliaksandr Valialkin
1b14cf18b6
lib/storage: ignore start text and end text anchors in getOrValues(regexp) function
...
This is OK, since the anchors are implicitly applied to the whole regexp.
This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors.
For example, `{label="^(foo|bar)$"}`
2022-08-24 17:16:28 +03:00
Aliaksandr Valialkin
7b9ba456ff
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
...
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:
vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll
...
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.
This is a follow-up for 02ca2342ab
2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
88e0fe9469
all: use os.ReadDir instead of ioutil.ReadDir
...
The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir
This is a follow-up for 02ca2342ab
2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:55:20 +03:00
Roman Khavronenko
fc2b8b4efd
lib/storage: bump max merge concurrency for small parts to 15 ( #2997 )
...
* lib/storage: bump max merge concurrency for small parts to 15
The change is based on the feedback from users on github.
Thier examples show, that limit of 8 sometimes become a
bottleneck. Users report that without limit concurrency
can climb up to 15-20 merges at once.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Update lib/storage/partition.go
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-21 23:37:14 +03:00
Aliaksandr Valialkin
1c7f402598
app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling
...
For example:
- target_label: composite-label
replacement: {{source_label1}}-{{source_label2}}
2022-08-21 22:49:24 +03:00
Roman Khavronenko
2c59c83191
lib/storage: fix the search for empty label name ( #2991 )
...
* lib/storage: fix the search for empty label name
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Apply suggestions from code review
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-19 11:05:09 +03:00
Aliaksandr Valialkin
1812d33a2d
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit
...
The following metrics are generated:
- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped
These metrics simplify alerting on targets, which expose too many time series
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin
aa37e6b438
lib/promscrape: retry http requests if the server returns 429 status code
...
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin
1a363192ff
lib/storage: typo fix in comments after f830edc0bc
2022-08-16 13:45:32 +03:00
Aliaksandr Valialkin
dc929e0d16
lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
2022-08-16 13:34:23 +03:00
Aliaksandr Valialkin
c27bd63f6c
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html
2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin
1a00c9ef03
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11034
2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin
2fb63dda83
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11002
2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin
2b58bd9876
lib/promscrape/discovery/dns: add support for resolving MX records
...
See https://github.com/prometheus/prometheus/pull/10099
2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin
10402459d8
lib/vmselectapi: do not log connection accept/close from vmselect
...
These log messages became too spammy in production clusters
after the commit 190c8b463chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508
2022-08-12 09:15:29 +03:00
Aliaksandr Valialkin
1b39be3305
lib/vmselectapi: add rpc call prefix to the trace of the rpc call in order to make it more clear
2022-08-12 00:20:49 +03:00
Roman Khavronenko
f42853275f
lib/storage: prevent excessive loops when storage is in RO ( #2962 )
...
* lib/storage: prevent excessive loops when storage is in RO
Returning nil error when storage is in RO mode results
into excessive loops and function calls which could
result into CPU exhaustion. Returning an err instead
will trigger delays in the for loop and save some resources.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* document the change
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-09 12:17:47 +03:00
Aliaksandr Valialkin
310779d8b5
lib/promscrape: follow-up after 2c553d5a2f
...
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
2022-08-08 14:49:16 +03:00
