Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-20 23:46:23 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,054 Commits 114 Branches 748 Tags 625 MiB
ba8ac08739
Commit Graph

159 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
fd9fd191b9 lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server 
Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg.
See https://www.consul.io/api-docs/health#list-nodes-for-service for details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170
 2020-12-08 21:52:42 +02:00
Aliaksandr Valialkin
364f30a6e7 lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork() 
This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice
could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they
are no longer used.

See the related commit e205975716 and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-12-08 17:50:05 +02:00
Aliaksandr Valialkin
08b71d2067 lib/promscrape: re-use strings for labels stored in ScrapeWork 
This should reduce memory usage when working with big number of scrape targets.
 2020-12-08 12:22:59 +02:00
Aliaksandr Valialkin
0f1b969aa6 lib/promscrape: export vm_promscrape_scrapers_{started|stopped}_total metrics for monitoring target churn rate 2020-12-08 11:57:52 +02:00
Aliaksandr Valialkin
c7ac7c1807 lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value 
This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.
 2020-12-08 11:50:48 +02:00
Aliaksandr Valialkin
05813259dc lib/promscrape: export vm_promscrape_active_scrapers{type="<sd_type>"} metric for tracking the number of active scrapers per each service discovery type 2020-12-08 01:54:23 +02:00
Aliaksandr Valialkin
9c1c9d8e76 lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun command-line flag is passed 
Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944
 2020-12-07 13:18:50 +02:00
Aliaksandr Valialkin
82972a8f2a lib/promscrape: mention in scrape error message that scrape errors can be disabled by -promscrape.suppressScrapeErrors command-line flag 2020-12-06 23:27:58 +02:00
Aliaksandr Valialkin
299a35948c lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set 2020-12-06 13:18:39 +02:00
Aliaksandr Valialkin
52915c8f7e lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries 
Too low limit could result in unexpected errors when performing big number of blocking queries.
 2020-12-05 12:15:52 +02:00
Aliaksandr Valialkin
de0643fab5 lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher 2020-12-03 20:14:55 +02:00
Aliaksandr Valialkin
9cd8eb92f1 lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header 2020-12-03 20:05:23 +02:00
Aliaksandr Valialkin
5009b25a03 lib/promscrape: code cleanup after c6dee6c52d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
 2020-12-03 19:50:53 +02:00
Nikolay
c6dee6c52d
Changes consul discovery api (#921) 
* adds consul watch api,
it must reduce load on consul service with blocking wait requests,
changed discoveryClient api with fetchResponseMeta callback.

* small fix

* fix after master merge

* adds watch client at discovery utils

* fixes consul watcher,
changes namings,
fixes data race

* small typo fix

* sanity fix

* fix naming and service node update
 2020-12-03 19:47:40 +02:00
Aliaksandr Valialkin
32869e4c0f lib/promscrape: fix failing tests after a906b3862f 2020-11-29 01:26:03 +02:00
Aliaksandr Valialkin
3a32789352 lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets 2020-11-26 18:32:06 +02:00
Aliaksandr Valialkin
2cea4d403f all: typo fix: thouthand->thousand 2020-11-26 13:33:46 +02:00
Aliaksandr Valialkin
b0a5c382ee lib/promscrape: release http response non-200 status code 2020-11-26 13:25:17 +02:00
Aliaksandr Valialkin
b7fcdb528d app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun 2020-11-25 22:59:13 +02:00
Aliaksandr Valialkin
aa90b93778 lib/promscrape: expose __meta_ec2_ipv6_addresses label for ec2_sd_config like Prometheus will do in the next release 2020-11-23 16:56:42 +02:00
Aliaksandr Valialkin
de523c81b9 lib/promscrape: add filters option to dockerswarm_sd_config like Prometheus did in v2.23.0 2020-11-23 16:27:40 +02:00
Aliaksandr Valialkin
9d160f9048 lib/promscrape: hint that -enableTCP6 command-line flag can be used for connecting to IPv6 addresses 2020-11-21 14:39:00 +02:00
Aliaksandr Valialkin
d7932775cc lib/promscrape/discovery/eureka: follow-up after eec76718e9 2020-11-20 14:00:12 +02:00
Nikolay
eec76718e9
Adds eureka service discovery (#913) 
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-20 13:38:12 +02:00
Vasily
6fcbd17bdd Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig (#796) 
* Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig

* Add omitempty annotation to all the default/optional values

* Fix annotations after review
 2020-11-13 16:19:05 +02:00
Aliaksandr Valialkin
e205975716 lib/promscrape: make a copy of ScrapeWork from discovered []ScrapeWork slice instead of referring to an item in this slice 
This should prevent from holding previously discovered []ScrapeWork slices when a part of discovered targets changes over time.
This should reduce memory usage for the case when big number of discovered scrape targets changes over time.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-10 16:13:05 +02:00
Aliaksandr Valialkin
6e668fd480 lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets 
This should reduce load on GC a bit when discovering big number of scrape targets
 2020-11-10 15:56:51 +02:00
Aliaksandr Valialkin
bcd139362b lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-10 00:19:57 +02:00
Aliaksandr Valialkin
6c24c5caa3 lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-09 10:54:42 +02:00
Aliaksandr Valialkin
83df20b5b5 lib/promscrape: clean references to label name and label value strings after applying per-target relabeling 
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-07 16:19:59 +02:00
Aliaksandr Valialkin
9e83335ca9 lib/promscrape/discovery/kubernetes: go fmt 2020-11-07 13:03:49 +02:00
Aliaksandr Valialkin
5407eed2f6 lib/promscrape/discovery/kubernetes: reduce memory usage for labels when discovering big number of scrape targets by using string concatenation instead of fmt.Sprintf 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-07 13:03:08 +02:00
Aliaksandr Valialkin
188325f0fc lib/promscrape: eliminate data race in stream parse mode 
Previously `-promscrape.streamParse` mode could result in garbage labels for the scraped metrics because of data race.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247
 2020-11-07 12:45:17 +02:00
Nikolay
4c808d58bf
Adds ready probe (#874) 
* adds leading forward slash check for scrapeURL path
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835

* adds ready probe for scrape config initialization,
it should prevent metrics loss during vmagent rolling update,
/ready api will return 425 http code, if some scrape config still waits for initialization.

* updates docs

* Update app/vmagent/README.md

* renames var

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 20:29:18 +02:00
Aliaksandr Valialkin
b2042a1c30 lib/promscrape: docs update after e4182dd896 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
 2020-11-04 17:12:30 +02:00
Nikolay
e4182dd896
reduces memory usage for vmagent, (#880) 
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878

* Update app/vmagent/README.md

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 17:03:43 +02:00
Aliaksandr Valialkin
b9e5172aa2 Revert "lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets" 
This reverts commit 5a9743211f.
 2020-11-04 11:45:02 +02:00
Aliaksandr Valialkin
5a9743211f lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
 2020-11-04 11:08:57 +02:00
Aliaksandr Valialkin
ca8b5745b5 lib/promscrape: reduce memory allocations in promLabelsString() function 
This should help with reducing memory usage in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
 2020-11-04 10:38:44 +02:00
Aliaksandr Valialkin
c0f3be824d lib/promscrape: properly handle response body after 301 redirect 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869
 2020-11-02 01:09:52 +02:00
Aliaksandr Valialkin
e277c3d07b lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics 2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
b7638f04a7 app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets 
This page is exposed by vmagent and by a single-node VictoriaMetrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
 2020-11-01 23:35:06 +02:00
Nikolay
67059caa12
fixes panic at scrape error body formating, (#868) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864
regression after body reuse improvements
 2020-10-29 17:17:52 +03:00
Nikolay
de3fe22815
adds leading forward slash check for scrapeURL path (#855) 
* fixes in-consistency with prometheus behaviour for scrape targets url path.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835
 2020-10-29 08:39:42 +03:00
Nikolay Khramchikhin
e6ab69dd88
fixes openstack api endpoint with suffix trim adds openstack (#840) 
api v2.0 check
 2020-10-16 21:20:57 +03:00
Aliaksandr Valialkin
63c4999e06 lib/promscrape: code prettifying after 9bd9f67718 2020-10-12 16:12:36 +03:00
Nikolay Khramchikhin
9bd9f67718
Adds dockerswarm sd (#818) 
* adds dockerswarm service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656

 Following roles supported: services, tasks and nodes.
 Basic, token and tls auth supported.
 Added tests for labels generation.

* added unix socket support to discovery utils

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-10-12 13:38:21 +03:00
Aliaksandr Valialkin
561a7619a5 lib/promscrape: fix tests after 71ea4935de 2020-10-08 19:32:36 +03:00
Aliaksandr Valialkin
71ea4935de lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors 
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.

Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
 2020-10-08 18:58:30 +03:00
Aliaksandr Valialkin
90aa2a8ffd lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human 2020-10-06 11:34:09 +03:00