Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-22 08:26:19 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6,149 Commits 114 Branches 748 Tags 649 MiB
b21a55febf
Commit Graph

578 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
f392913d00
lib/promscrape: follow-up after bced9fb978 
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-05 21:13:06 -08:00
Zakhar Bessarab
bced9fb978
lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568) 2023-01-05 19:34:47 -08:00
Aliaksandr Valialkin
9f348cf8a1
lib/promscrape/discovery/nomad: follow-up after 48f371a46c 
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 18:07:20 -08:00
Aliaksandr Valialkin
1a28f0e5b3
lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 
This allows copy-n-pasting the url to another browser window and seeing the same result.

The limit in 1000 chars is selected in order to prevent from potential issues with systems
which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string

If the limit is exceeded, then query args are sent via POST method and aren't visible in the url.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 16:48:04 -08:00
Karan Sharma
48f371a46c
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549) 
Add nomad_sd_config support for service discovery
 2023-01-05 23:03:58 +01:00
Zakhar Bessarab
185cdcd813
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586) 
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 03:34:25 -08:00
Aliaksandr Valialkin
0dea3b71da
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2023-01-05 03:26:49 -08:00
Aliaksandr Valialkin
a1076abcbf
lib/promscrape: follow-up for a7e29c38bc 
- Document the bugfix at docs/CHANGELOG.md
- Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 02:52:08 -08:00
Zakhar Bessarab
a7e29c38bc
lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595) 
* lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/targetstatus: address review comment

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 02:39:31 -08:00
Aliaksandr Valialkin
5bca3a5be2
app/vmselect: remove dependency on lib/promscrape from app/vmselect 2023-01-03 23:28:27 -08:00
Aliaksandr Valialkin
add2c4bf07
lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..)) 2023-01-03 22:16:22 -08:00
Aliaksandr Valialkin
c4229a1bba
lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize 
This is a follow-up for 7ad9fff7e5
 2022-12-28 14:42:11 -08:00
Clément Nussbaumer
7ad9fff7e5
fix(promscrape): check MaxScrapeSize after gzip decompression (#3550) 2022-12-28 12:19:41 -08:00
Aliaksandr Valialkin
0076422350
lib/promscrape/discovery/azure: typo fix 2022-12-21 21:25:16 -08:00
Aliaksandr Valialkin
0bf3ae9559
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname> 
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
 2022-12-19 13:08:11 -08:00
Aliaksandr Valialkin
65f8fc527f
lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page 2022-12-16 17:02:41 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures 
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
 2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
a30ae502ef
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2022-12-10 12:44:45 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging 
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
 2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
2406c0dcfd
docs/CHANGELOG.md: document the bugfix at 05b42601c3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
 2022-12-08 18:35:28 -08:00
Zakhar Bessarab
05b42601c3
lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403) 
* lib/promscrape/discovery/azure: remove API server from URL returned by azure

* lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer
 2022-12-08 18:29:10 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
 2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
14660d4df5
all: typo fix: the the -> the 2022-12-03 21:53:01 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed 
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
 2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
295c84df66
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels 2022-11-29 20:27:48 -08:00
Aliaksandr Valialkin
654e94f420
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
 2022-11-28 18:37:09 -08:00
Roman Khavronenko
03d88bc066
vmagent: expose metrics for tracking config state (#3375) 
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-11-22 00:38:43 +02:00
Aliaksandr Valialkin
95f0266558
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones 
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
 2022-11-21 22:32:05 +02:00
Aliaksandr Valialkin
5955d23232
lib/promscrape: add a benchmark for internLabelStrings() 2022-11-16 23:02:49 +02:00
Aliaksandr Valialkin
86bce7f5f9
lib/promscrape: add more cases to TestAddRowToTimeseries 
This is a follow-up for 16fdd2af8a
 2022-11-09 16:13:56 +02:00
Jeremy PLANCKEEL
16fdd2af8a
test(golang): add test to function addRowToTimeseries (#3282) 
Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>
 2022-11-09 15:41:26 +02:00
Aliaksandr Valialkin
7fa5d043f5
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/11482
 2022-11-07 15:25:53 +02:00
Aliaksandr Valialkin
76e8888272
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set. 
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.

While at it, properly handle some corner cases:

- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278

Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
 2022-10-28 22:14:26 +03:00
Aliaksandr Valialkin
454baf84d6
lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page 2022-10-28 22:14:25 +03:00
Aliaksandr Valialkin
518c340ae3
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax 
This is a follow-up for 02096e06d0
 2022-10-26 14:49:33 +03:00
Nikolay
1059c4d84a
lib/promscrape/discovery/kubernetes: correctly wrap error (#3250) 
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-10-18 20:37:42 +03:00
Aliaksandr Valialkin
069401a304
all: log error when environment variables referred from -promscrape.config are missing 
This should prevent from using incorrect config files
 2022-10-18 10:47:16 +03:00
Aliaksandr Valialkin
50f5eae0e0
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places 
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.

This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
 2022-10-09 14:51:16 +03:00
Aliaksandr Valialkin
5269b1ad77
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis 
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
 2022-10-07 23:36:14 +03:00
Aliaksandr Valialkin
f9df0cae16
lib/promscrape: allow specifying full target url in __address__ label 
Previously the `__address__` label could contain only `host:port` part of the target url,
while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__`
labels. Now it is possible to set the full url in `__address__` label.

This makes valid the following scrape config, which is frequently used by novice users:

scrape_configs:
- job_name: foo
  static_configs:
  - targets:
    - http://host1/metrics1
    - https://host2/metrics2
 2022-10-07 22:43:04 +03:00
Aliaksandr Valialkin
4998402004
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics 
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
 2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin
f86070169d
lib/promscrape/discovery/azure: remove unneeded conversion to string 2022-10-01 16:04:37 +03:00
Aliaksandr Valialkin
c54e14cdec
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does 
See https://github.com/prometheus/prometheus/pull/11326
 2022-09-30 20:48:32 +03:00
Aliaksandr Valialkin
17289ff481
lib/regexutil: cache MatchString results for unoptimized regexps 
This increases relabeling performance by 3x for unoptimized regexs
 2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil 
lib/bytesutil is more appropriate place for InternString() function
 2022-09-30 07:44:35 +03:00
Aliaksandr Valialkin
6a32a64073
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed 2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin
56ce7ce85b
lib/promscrape: typo fix after 74c00a8762 2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin
74c00a8762
lib/promscrape: read response body into memory in stream parsing mode before parsing it 
This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
 2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin
ccad651a61
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error 
This should improve debugging issues with Kubernetes API server
 2022-09-13 19:36:55 +03:00
Bryce Lampe
74f8e12e87
Support "HTTP" and "HTTPS" schemes (#3019) 
* Support "HTTP" and "HTTPS" schemes

* Update lib/promscrape/config.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin
30b8d91727
lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache 
This is a follow-up for c06e7a142c
 2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName() 
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
 2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
1f89278d88
all: subsitute ioutil.ReadAll with io.ReadAll 
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
 2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
7d26414b2e
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit 
The following metrics are generated:

- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped

These metrics simplify alerting on targets, which expose too many time series

See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
 2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin
bb68ab99fa
lib/promscrape: retry http requests if the server returns 429 status code 
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
 2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin
c3f8481011
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html 2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin
95d36da358
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11034
 2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin
c4fcd9f1c5
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11002
 2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin
511805d88d
lib/promscrape/discovery/dns: add support for resolving MX records 
See https://github.com/prometheus/prometheus/pull/10099
 2022-08-15 00:32:34 +03:00
Aliaksandr Valialkin
46d7792b72
lib/promscrape: follow-up after 2c553d5a2f 
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
 2022-08-08 14:46:26 +03:00
Fury
2c553d5a2f
add support to scrape multi tenant metrics (#2950) 
* add support to scrape multi tenant metrics

* add support to scrape multi tenant metrics

Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
 2022-08-08 14:10:18 +03:00
Aliaksandr Valialkin
0ef29ceb14
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress 
See 7e65ad3e43
and 7e1111ff14
 2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin
f2816ef031
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs 
This option was ignored since d289ecded1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
 2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin
3e8890e71b
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels 
See https://github.com/prometheus/prometheus/issues/9187
 2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin
68de1f4e4a
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default 
This aligns with Prometheus behaviour.

See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
 2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin
02de848c88
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f 2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin
83a4abda3f
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
 2022-08-04 22:26:43 +03:00
Igor Tiunov
6e5ac32fba
YC service discovery (#2923) 
* YC service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386

* Fixed linter suggestions

* fixed golint errors
 2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin
4c2f9a1a2e
lib/promscrape: set up=0 for partially failed scrape in stream parsing mode 
This behaviour aligns with Prometheus behavior
 2022-07-22 13:29:44 +03:00
Aliaksandr Valialkin
0fd86e2364
lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
 2022-07-18 17:15:07 +03:00
Nikolay
7301aa678c
lib/promscrape: adds azure service discovery (#2743) 
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-13 23:43:18 +03:00
Aliaksandr Valialkin
f97355d9fb
lib/promscrape: properly set Host header when sending requests via http proxy 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
 2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin
01f55bc66b
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
 2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin
b186b63e07
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/10759
 2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin
e6ba2af7a1
lib/promscrape: fix a test after c66f676f3b 2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin
c66f676f3b
lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target 
See https://github.com/VictoriaMetrics/operator/issues/497
 2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin
855436efd2
lib/promauth: refactor NewConfig in order to improve maintainability 
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options

This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
 2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin
e40b40afe6
Revert "lib/promscrape, vmagent: fix path to files (#2801)" 
This reverts commit 0a8e35835c.

Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005
 2022-06-30 18:23:56 +03:00
Dmytro Kozlov
0a8e35835c
lib/promscrape, vmagent: fix path to files (#2801) 
vmagent: respect `-pathPrefix` flag for static files and links
 2022-06-30 16:22:54 +02:00
ttyv
bdf9f4669a
lib/promscrape: fix vmagent tickerCh reload behaviour (#2786) 
Co-authored-by: Dmitriy <dab@ttyv.ru>
 2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin
52eadb729e
lib/promscrape: always send stale markers with the real scrape timestamp 
This guarantees that query won't return data just after the series is disappeared.
 2022-06-23 11:34:18 +03:00
Aliaksandr Valialkin
1c4f67c5d2
lib/promauth: add ability to send additional http headers in requests to scrape targets 
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
 2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin
e6ed92529b
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This package already has the same name, so there is no need in explicit name
 2022-06-21 20:23:32 +03:00
Roman Khavronenko
63b538ecd1
vmagent: update SD duration histogram metric if SD is active (#2677) 
The change updates histogram for registering SD update duration
only SD is considered as `active`. SD is active if at least
one scraper for this SD has started.

This change supposed to reduce metrics cardinality produced
by duration histogram which gets updated even if SD isn't configured.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-06-07 15:46:44 +03:00
Aliaksandr Valialkin
a5814fe16a
lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string 
This improves code readability and maintainability a bit, since the format string
is passed as string literal into fmt.Errorf.
 2022-06-07 01:22:07 +03:00
Aliaksandr Valialkin
8608dd093c
all: follow-up after 8edb390e21 
- Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb
- Add /service-discovery page for API compatibility with Prometheus
- Properly load bootstrap.min.css from /prometheus/targets
- Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there
 2022-06-07 00:57:09 +03:00
Aliaksandr Valialkin
6f0a0e3072
lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534 
- make more clear error logs
- simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct
 2022-06-06 14:40:52 +03:00
Aliaksandr Valialkin
0922ed2b7e
lib/promscrape: add -promscrape.cluster.name command-line flag 
This flag is used for proper data de-duplication when the same target is scraped
from multiple vmagent clusters.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679
 2022-06-04 00:37:01 +03:00
Dmytro Kozlov
8edb390e21
lib/promscrape: adds service discovery visualization for /targets page(#2675) 
* lib/promscrape: updated template

* lib/promscrape: fixed click on unhealthy and all btns

* app/vmselect: jquery scripts into static folder

Co-authored-by: f41gh7 <nik@victoriametrics.com>
 2022-06-03 15:38:45 +02:00
Nikolay
a18914abee
lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 (#2672) 2022-06-01 20:44:45 +02:00
hadesy
006b8c7534
promscrape/discovery: support kubeconfig (#2533) 2022-06-01 20:34:00 +02:00
Aliaksandr Valialkin
d2567ccdd6
lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum 
In this case there is no need in converting int64 to int
 2022-06-01 01:42:34 +03:00
Dmytro Kozlov
11f91532c5
issue-2594: use embedded for static files (#2650) 
embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries.

Co-authored-by: f41gh7 <nik@victoriametrics.com>

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594
 2022-05-31 01:55:28 +02:00
Dmytro Kozlov
1eb29794e6
removed redundant return (fixed linter) (#2647) 
* removed redundant return

* updated lint package version
 2022-05-26 16:24:01 +02:00
Aliaksandr Valialkin
796804e4b0
lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag 
This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575

The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 .
Thanks to @jelmd .
 2022-05-25 22:59:36 +03:00
Roman Khavronenko
d5eb6afe26
lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) 
* lib/promscrape/discovery/kubernetes: properly updates discovered scrape works
previously, added or updated scrapeworks may override previuosly
discovered.
it happens because swosByKey may contain small subset of kubernetes
objects with it's labels.
It happens for objectsUpdated and objectsAdded maps, which include only changed elements

* Properly calculate vm_promscrape_discovery_kubernetes_scrape_works

Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-21 01:01:37 +03:00
Boris Petersen
3df8caca15
Add ability to sign requests for all AWS services (#2604) 
This adds the ability to utilize sigv4 signing for all AWS services not
just "aps". When the newly introduced property "service" is not set it
will default to "aps".

Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
 2022-05-18 14:58:31 +02:00
Aliaksandr Valialkin
a0727ab1b1
docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag 2022-05-12 18:50:29 +03:00
Aliaksandr Valialkin
9ea3f0c0d3
lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future 
Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.
 2022-05-09 15:33:22 +03:00