Aliaksandr Valialkin
02c0959380
lib/promscrape: make cluster membership calculations consistent across 32-bit and 64-bit architectures
2021-03-05 09:06:08 +02:00
Aliaksandr Valialkin
133fb9fc00
lib/promscrape: add -promscrape.cluster.replicationFactor
command-line flag for replicating scrape targets among vmagent
instances in the cluster
2021-03-04 10:21:27 +02:00
Aliaksandr Valialkin
fea27b9845
lib/promscrape: go fmt
2021-03-02 21:20:23 +02:00
Aliaksandr Valialkin
3fbe2bf1c8
lib/promscrape: pre-allocate space for a map in mergeLabels
...
This should reduce the number of memory allocations when discovering big number of targets
2021-03-02 18:42:44 +02:00
Aliaksandr Valialkin
f9c1fe3852
lib/promscrape/discovery/kubernetes: cache ScrapeWork objects as soon as the corresponding k8s objects are changed
...
This should reduce CPU usage and memory usage when Kubernetes contains tens of thousands of objects
2021-03-02 16:44:19 +02:00
Aliaksandr Valialkin
c3bf72992f
lib/promscrape: explicitly stop and cleanup service discovery routines when new config is read from -promscrape.config
...
This should reduce memory usage when `-promscrape.config` file frequently changes
2021-03-01 14:15:16 +02:00
Aliaksandr Valialkin
8af9370bf2
lib/promscrape: use target arg in ScrapeWork cache
2021-03-01 12:31:11 +02:00
Aliaksandr Valialkin
5328506c3e
lib/promscrape: typo fix, which prevented from caching ScrapeWork entries
2021-03-01 12:13:13 +02:00
Aliaksandr Valialkin
d5058caccc
lib/promscrape: add vm_promscrape_scrapework_cache_* metrics for tracking ScrapeWork cache effectiveness
2021-03-01 12:05:58 +02:00
Aliaksandr Valialkin
109bfaadad
lib/promscrape: reduce CPU usage an memory allocations when constructing scrapeWorkKey
2021-02-28 22:30:36 +02:00
Aliaksandr Valialkin
9a2bf65134
lib/promscrape: add ability to spread scrape targets among multiple vmagent instances
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
2021-02-28 18:40:42 +02:00
Aliaksandr Valialkin
6f8866a40a
lib/promscrape: fix possible deadlock in parallel execution of target relabeling
2021-02-28 16:05:32 +02:00
Aliaksandr Valialkin
44975e28fe
lib/promscrape: yet another typo fix after ed8441ec52
2021-02-26 23:36:34 +02:00
Aliaksandr Valialkin
50e74c439c
lib/promscrape: typo fix after ed8441ec52
2021-02-26 23:04:40 +02:00
Aliaksandr Valialkin
fa3ce450fb
lib/promscrape: cache ScrapeWork
...
This should reduce the time needed for updating big number of scrape targets.
2021-02-26 21:43:41 +02:00
Aliaksandr Valialkin
dc8c045378
lib/promscrape: cleanup after 9b2246c29b
...
Main points:
* Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit
* Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001
* Corner case fixes.
2021-02-26 19:09:12 +02:00
Nikolay
cf9262b01f
vmagent kubernetes watch stream discovery. ( #1082 )
...
* started work on sd for k8s
* continue work on watch sd
* fixes
* continue work
* continue work on sd k8s
* disable gzip
* fixes typos
* log errror
* minor fix
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin
4cfac70cde
lib/promscrape: remove duplicate code a bit
2021-02-26 15:54:07 +02:00
Aliaksandr Valialkin
55eb983193
lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel
2021-02-26 12:48:48 +02:00
Aliaksandr Valialkin
197ecca426
lib/promrelabel: add more optimizations for relabeling for common cases
2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin
8b87398333
lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric
2021-02-21 23:21:35 +02:00
Aliaksandr Valialkin
502d0e2524
lib/promscrape: add scrape_align_interval config option into scrape config
...
This option allows aligning scrapes to a particular intervals.
2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin
367fc17933
lib/promscrape: code prettifying for 8dd03ecf19
2020-12-24 10:57:20 +02:00
Nikolay
b00f7816e2
adds proxy_url support, ( #980 )
...
* adds proxy_url support,
adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery,
adds proxy_url to the scrape_config for targets scrapping,
http based proxy is supported atm,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503
* fixes imports
2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin
1ee5a234dc
lib/promscrape: remove ID
field from ScrapeWork
struct. Use a pointer to ScrapeWork as a key in targetStatusMap
...
This simplifies the code a bit.
2020-12-17 14:31:55 +02:00
Aliaksandr Valialkin
38188e1d6b
lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork()
...
This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice
could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they
are no longer used.
See the related commit e205975716
and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-12-08 17:55:21 +02:00
Aliaksandr Valialkin
d5faad0240
lib/promscrape: re-use strings for labels stored in ScrapeWork
...
This should reduce memory usage when working with big number of scrape targets.
2020-12-08 12:23:44 +02:00
Aliaksandr Valialkin
b5b32c65b0
lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun
command-line flag is passed
...
Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944
2020-12-07 13:18:16 +02:00
Aliaksandr Valialkin
3f52e59efe
app/{vmagent,victoria-metrics}: add -dryRun
option and make more clear handling for -promscrape.config.dryRun
2020-11-25 23:01:39 +02:00
Nikolay
bb2bcb9725
Adds eureka service discovery ( #913 )
...
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-20 14:02:13 +02:00
Vasily
8ba168f3be
Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig ( #796 )
...
* Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig
* Add omitempty annotation to all the default/optional values
* Fix annotations after review
2020-11-13 16:17:03 +02:00
Aliaksandr Valialkin
2ec02b7bdb
lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets
...
This should reduce load on GC a bit when discovering big number of scrape targets
2020-11-10 15:57:43 +02:00
Aliaksandr Valialkin
a8562d643b
lib/promscrape: add -promscrape.dropOriginalLabels
command-line flag for reducing memory usage when discovering big number of scrape targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin
aa3e46a02d
lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-09 10:55:03 +02:00
Aliaksandr Valialkin
b8083b7659
lib/promscrape: clean references to label name and label value strings after applying per-target relabeling
...
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin
2cd86d0220
lib/promscrape: docs update after e4182dd896
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
2020-11-04 17:13:34 +02:00
Nikolay
d0a9b24c5a
reduces memory usage for vmagent, ( #880 )
...
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
* Update app/vmagent/README.md
* Update app/vmagent/README.md
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin
f35aafb6a5
Revert "lib/promscrape: add -promscrape.dropOriginalLabels
command-line flag for reducing memory usage when discovering big number of scrape targets"
...
This reverts commit b08c6f5144
.
2020-11-04 11:45:38 +02:00
Aliaksandr Valialkin
b08c6f5144
lib/promscrape: add -promscrape.dropOriginalLabels
command-line flag for reducing memory usage when discovering big number of scrape targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
2020-11-04 11:09:05 +02:00
Nikolay
058f49de57
adds leading forward slash check for scrapeURL path ( #855 )
...
* fixes in-consistency with prometheus behaviour for scrape targets url path.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835
2020-11-01 23:22:16 +02:00
Aliaksandr Valialkin
ed724d25ba
lib/promscrape: add stream parse
mode for efficient scraping of targets that expose millions of metrics
2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
abdf22e0bb
app/vmagent: expose /api/v1/targets
page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets
...
This page is exposed by vmagent and by a single-node VictoriaMetrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
2020-10-20 21:55:14 +03:00
Nikolay Khramchikhin
7f96712b38
Adds dockerswarm sd ( #818 )
...
* adds dockerswarm service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656
Following roles supported: services, tasks and nodes.
Basic, token and tls auth supported.
Added tests for labels generation.
* added unix socket support to discovery utils
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin
0d44e371f3
lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors
command-line flag in order to suppress duplicate scrape target
errors
...
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.
Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
2020-10-08 18:59:25 +03:00
Nikolay Khramchikhin
b4c77fc6d2
Adds openstack sd ( #811 )
...
* adds openstack service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728
implemented hypervisors and instance discovery with openstack v3 api.
Added tests for labeling and data parsing.
Added token refresh.
* Apply suggestions from code review
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin
60c7397be5
all: support %{ENV_VAR}
placeholders in yaml configs in all the vm* components
...
Such placeholders are substituted by the corresponding environment variable values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
2361ad8ab4
lib/promscrape: add ability to set disable_compression
and disable_keepalive
options in scrape_config
section of the config passed to -promscrape.config
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
2020-07-02 14:19:34 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
de7e585ac8
lib/promscrape: preserve the previously discovered targets on discovery errors per each job_name
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582
2020-06-23 15:42:46 +03:00
Aliaksandr Valialkin
482bae8466
lib/promscrape: add -promscrape.config.dryRun
flag for checking -promscrape.config
for errors or unsupported options
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 14:54:32 +03:00