Aliaksandr Valialkin
4eb9926125
lib/promscrape: code cleanup: send stale markers immediately after generating automatic metrics
...
This cleanup has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557/files#diff-6b205cf6637d7b65a5c45d9417d08822d4efad94227268cb196f61aa2a0fc0f7
2024-01-21 05:18:22 +02:00
Aliaksandr Valialkin
12f2c5679b
all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually
2024-01-21 05:11:05 +02:00
Aliaksandr Valialkin
90768aa418
lib/storage/partition.go: remove misleading comment, which falsely states that inmemoryParts isn't visible to search
...
Thanks to @satjd for raising attention to this comment at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5410
2024-01-21 04:49:35 +02:00
Aliaksandr Valialkin
7fba73ce11
lib/promscrape/discovery/kubernetes: add -promscrape.kubernetes.attachNodeMetadataAll command-line flag
...
This flag allows setting attach_metadata.node=true for all the kubernetes_sd_configs defined at -promscrape.config
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640
Thanks to wasim-nihal for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593
2024-01-21 03:13:56 +02:00
Nikolay
8ab0ce3ded
app/vmselect: abort streaming connections for vmselect ( #5650 )
...
* app/vmselect: abort streaming connections for vmselect
due to streaming nature of export APIs, curl and simmilr tools cannot
detect errors that happened after http.Header with status 200 was
written to it.
This PR tracks if body write was already started and closes connection.
It allows client to detect not expected chunk sequence and return error
to the caller.
Mostly it affects vmselect at cluster version
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645
* wip
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5650
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 02:12:51 +02:00
Aliaksandr Valialkin
74448a7e57
lib/promscrape/discovery/hetzner: follow-up after 03a97dc678
...
- docs/sd_configs.md: moved hetzner_sd_configs docs to the correct place according to alphabetical order of SD names,
document missing __meta_hetzner_role label.
- lib/promscrape/config.go: added missing MustStop() call for Hetzner SD,
and moved the code to the correct place according to alphabetical order of SD names.
- lib/promscrape/discovery/hetzner: properly handle pagination for hloud API responses,
populate missing __meta_hetzner_role label like Prometheus does.
- Properly populate __meta_hetzner_public_ipv6_network label like Prometheus does.
- Remove unused SDConfig.Token.
- Remove "omitempty" annotation from SDConfig.Role field, since this field is mandatory.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154
2024-01-20 17:01:53 +02:00
Hui Wang
cfec258803
lib/promscrape: do not store last scrape response when stale markers … ( #5577 )
...
* lib/promscrape: do not store last scrape response when stale markers are disabled
* update changelog
2024-01-20 00:53:41 +08:00
Aliaksandr Valialkin
5bdf62de5b
lib/storage: do not prefetch metric names for small number of metricIDs
...
This eliminates prefetchedMetricIDsLock lock contention for queries, which return less than 500 time series.
This is a follow-up for 9d886a2eb0
2024-01-17 13:48:15 +02:00
Aliaksandr Valialkin
41932db848
lib/promscrape: cosmetic changes after 3ac44baebe
...
- Rename mustLoadScrapeConfigFiles() to loadScrapeConfigFiles(), since now it may return error.
- Split too long line with the error message into two lines in order to improve readability a bit.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5508
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5560
2024-01-16 22:29:09 +02:00
Aliaksandr Valialkin
4073bb3303
lib/httputils: handle step=undefined query arg as an empty value
...
This is needed for Grafana, which may send step=undefined
when working with alerting rules and instant queries.
2024-01-16 18:59:26 +02:00
Aliaksandr Valialkin
9d886a2eb0
lib/storage: follow-up for 4b8088e377
...
- Clarify the bugfix description at docs/CHANGELOG.md
- Simplify the code by accessing prefetchedMetricIDs struct under the lock
instead of using lockless access to immutable struct.
This shouldn't worsen code scalability too much on busy systems with many CPU cores,
since the code executed under the lock is quite small and fast.
This allows removing cloning of prefetchedMetricIDs struct every time
new metric names are pre-fetched. This should reduce load on Go GC,
since the cloning of uin64set.Set struct allocates many new objects.
2024-01-16 15:29:57 +02:00
Hui Wang
3ac44baebe
exit vmagent if there is config syntax error in scrape_config_files when -promscrape.config.strictParse=true ( #5560 )
2024-01-16 17:30:02 +08:00
Aliaksandr Valialkin
388d020b7c
app/vmselect/promql: follow-up for ce4f26db02
...
- Document the bugfix at docs/CHANGELOG.md
- Filter out NaN values before sorting as suggested at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509#discussion_r1447369218
- Revert unrelated changes in lib/filestream and lib/fs
- Use simpler test at app/vmselect/promql/exec_test.go
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506
2024-01-16 02:55:08 +02:00
Zongyang
ce4f26db02
FIX bottomk doesn't return any data when there are no time range overlap between timeseries ( #5509 )
...
* FIX sort order in bottomk
* Add lessWithNaNsReversed for bottomk
* Add ut for TopK
* Move lt from loop
* FIX lint
* FIX lint
* FIX lint
* Mod log format
---------
Co-authored-by: xiaozongyang <xiaozngyang@kanyun.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-16 02:19:36 +02:00
Aliaksandr Valialkin
bfa73ebdf3
lib/prompbmarshal: move WriteRequest proto definition to the correct place
2024-01-16 00:20:37 +02:00
Aliaksandr Valialkin
4b42c8abbb
lib/promscrape/discovery/hetzner: fix golangci-lint warnings after 03a97dc678
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550
2024-01-15 17:12:40 +02:00
Aliaksandr Valialkin
be509b3995
lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop()
...
Previously the was a race condition when the background goroutine still could try collecting metrics
from already stopped resources after returning from pushmetrics.Stop().
Now the pushmetrics.Stop() waits until the background goroutine is stopped before returning.
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549
and the commit fe2d9f6646https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548
2024-01-15 13:50:36 +02:00
Aleksandr Stepanov
03a97dc678
vmagent: added hetzner sd config ( #5550 )
...
* added hetzner robot and hetzner cloud sd configs
* remove gettoken fun and update docs
* Updated CHANGELOG and vmagent docs
* Updated CHANGELOG and vmagent docs
---------
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-01-15 10:13:22 +01:00
Roman Khavronenko
4b8088e377
lib/storage: properly check for storage/prefetchedMetricIDs cache expiration deadline ( #5607 )
...
Before, this cache was limited only by size.
Cache invalidation by time happens with jitter to prevent thundering herd problem.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-15 10:03:06 +01:00
Aliaksandr Valialkin
d2c94a0663
lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto
2024-01-14 23:04:45 +02:00
Aliaksandr Valialkin
c005245741
lib/prompb: switch to github.com/VictoriaMetrics/easyproto
2024-01-14 22:46:06 +02:00
Aliaksandr Valialkin
f2229c2e42
lib/prompb: change type of Label.Name and Label.Value from []byte to string
...
This makes it more consistent with lib/prompbmarshal.Label
2024-01-14 22:33:21 +02:00
Aliaksandr Valialkin
f405384c8c
lib/protoparser/datadogv2: simplify code for parsing protobuf messages after 0597718435
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451
2024-01-14 21:43:01 +02:00
Aliaksandr Valialkin
dd25049858
lib/protoparser/opentelemetry: use github.com/VictoriaMetrics/easyproto for protobuf message unmarshaling and marshaling
...
This reduces VictoriaMetrics binary size by 100KB.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2424
2024-01-14 21:19:03 +02:00
Aliaksandr Valialkin
0597718435
lib/protoparser/datadogv2: add support for reading protobuf-encoded requests at /api/v2/series endpoint
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094
2024-01-14 21:09:05 +02:00
Artem Navoiev
d374595e31
docs: mention staleNaN handling during deduplication
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5587
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-11 11:53:58 +01:00
hagen1778
91ccea236f
app/all: follow-up after 84d710beab
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-09 13:34:54 +01:00
zhdd99
fe2d9f6646
lib/pushmetrics: fix a panic caused by pushing metrics during the graceful shutdown process of vmstorage nodes. ( #5549 )
...
Co-authored-by: zhangdongdong <zhangdongdong@kuaishou.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-09 13:24:34 +01:00
Aliaksandr Valialkin
7575f5c501
lib/protoparser/datadogv2: take into account source_type_name field, since it contains useful value such as kubernetes, docker, system, etc.
2023-12-21 23:05:41 +02:00
Aliaksandr Valialkin
b4ba8d0d76
lib/protoparser: add missing /datadog/ prefix to the /api/v2/series path in the description for -datadog.maxInsertRequestSize command-line flag
2023-12-21 21:04:53 +02:00
Aliaksandr Valialkin
fb90a56de2
app/{vminsert,vmagent}: preliminary support for /api/v2/series ingestion from new versions of DataDog Agent
...
This commit adds only JSON support - https://docs.datadoghq.com/api/latest/metrics/#submit-metrics ,
while recent versions of DataDog Agent send data to /api/v2/series in undocumented Protobuf format.
The support for this format will be added later.
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451
2023-12-21 20:50:55 +02:00
Aliaksandr Valialkin
01f9edda64
lib/promauth: add more context to errors returned by Options.NewConfig() in order to simplify troubleshooting
2023-12-20 21:58:12 +02:00
Aliaksandr Valialkin
160cc9debd
app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags
...
This is a follow-up for 5ebd5a0d7bhttps://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427
2023-12-20 21:35:28 +02:00
Morgan
5ebd5a0d7b
Expose OAuth2 Endpoint Parameters to cli ( #5427 )
...
The user may which to control the endpoint parameters for instance to
set the audience when requesting an access token. Exposing the
parameters as a map allows for additional use cases without requiring
modification.
2023-12-20 20:16:43 +02:00
Nikolay
7cfde237ec
lib/awsapi: properly assume role with webIdentity token ( #5495 )
...
* lib/awsapi: properly assume role with webIdentity token
introduce new irsaRoleArn param for config. It's only needed for authorization with webIdentity token.
First credentials obtained with irsa role and the next sts assume call for an actual roleArn made with those credentials.
Common use case for it - cross AWS accounts authorization
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822
* wip
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-12-20 19:05:39 +02:00
Aliaksandr Valialkin
5a88bc973f
all: use Gauge instead of Counter for *_config_last_reload_successful metrics
...
This allows exposing the correct TYPE metadata for these labels when the app runs with -metrics.exposeMetadata command-line flag.
See https://github.com/VictoriaMetrics/metrics/pull/61#issuecomment-1860085508 for more details.
This is follow-up for 326a77c697
2023-12-20 14:23:42 +02:00
Aliaksandr Valialkin
326a77c697
all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page
...
This may be needed for systems, which require this metadata such as Google Cloud Managed Prometheus.
See https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
2023-12-19 03:20:40 +02:00
Aliaksandr Valialkin
4b529562ce
lib/pushmetrics: add -pushmetrics.header and -pushmetrics.disableCompression command-line flags
2023-12-17 19:56:46 +02:00
Aliaksandr Valialkin
0379a0eb82
lib/protoparser/opentelemetry: allow ingesting metrics without resource labels
...
Some clients may ingest samples via OpenTelemetry protocol without Resource labels.
Previously VictoriaMetrics was silently dropping such samples.
The commit 317834f876https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5459
It is better from usability PoV to accept such samples instead of dropping them and incrementing the corresponding counter.
2023-12-17 19:12:58 +02:00
Zakhar Bessarab
317834f876
lib/protoparser/opentelemetry: add metric to track skipped rows without resource ( #5459 )
...
Currently, it is impossible to understand why metrics are not ingested when resource is not set by OTEL exporter. Adding metric should simplify debugging and make it improve debuggability.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-12-15 11:16:25 +01:00
Aliaksandr Valialkin
72dbd24b22
lib/fs: remove unused IsEmptyDir()
...
This function became unused after the commit 43b24164efhttps://go.dev/blog/deadcode
2023-12-14 19:38:53 +02:00
Aliaksandr Valialkin
0f91f83639
app/vmselect: add support for vmstorage groups with independent -replicationFactor per group
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197
See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmstorage-groups-at-vmselect
Thanks to @zekker6 for the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/718
2023-12-13 00:14:45 +02:00
hagen1778
e0fc5ef140
lib/promscrape: comsetic changes after e373bb84d5
...
* fix typos in docs
* add `shard-` prefix to generated links when `-promscrape.cluster.memberURLTemplate` is enabled
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-12-12 11:28:18 +01:00
Aliaksandr Valialkin
51df2248f0
vendor: run make vendor-update
2023-12-11 10:48:36 +02:00
Aliaksandr Valialkin
042267541f
app/vmauth: add support for hot standby mode via first_available load balancing policy
...
vmauth in `hot standby` mode sends requests to the first url_prefix while it is available.
If the first url_prefix becomes unavailable, then vmauth falls back to the next url_prefix.
This allows building highly available setup as described at https://docs.victoriametrics.com/vmauth.html#high-availability
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792
2023-12-08 23:31:07 +02:00
Aliaksandr Valialkin
b05e1512d4
lib/promscrape: add a wraning when the /service-discovery page contains incomplete list of dropped targets
2023-12-08 19:03:51 +02:00
noodles2hg
8efe694160
lib/streamaggr/streamaggr.go: fix link in error message ( #5439 )
2023-12-08 16:55:05 +03:00
Aliaksandr Valialkin
e373bb84d5
lib/promscrape: add -promscrape.cluster.memberURLTemplate command-line flag for creating direct links to vmagent instances at /service-discovery page
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018#issuecomment-1843811569
2023-12-07 16:04:21 +02:00
Aliaksandr Valialkin
7cb8ed8271
lib/promscrape: show -promscrape.cluster.memberNum values for vmagent instances, which scrape the given dropped target at /service-discovery page
...
The /service-discovery page contains the list of all the discovered targets
after the commit 487f6380d0https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets ).
This commit improves debuggability of targets in cluster mode by providing a list of -promscrape.cluster.memberNum
values per each target at /service-discovery page, which has been dropped becasue of sharding,
e.g. if this target is scraped by other vmagent instances in the cluster.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018
2023-12-07 00:05:32 +02:00
Aliaksandr Valialkin
67468a0c46
lib/promscrape: show never scraped message for never scraped targets at /targets page
2023-12-06 22:33:39 +02:00
