Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 08:23:34 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
3,143 Commits 114 Branches 748 Tags 504 MiB
09670479cd
Commit Graph

342 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
09670479cd lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric 
Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics.

The main idea is to perform the calculations on scrape_series_added and series_limit
only if the set of metrics exposed by the target has been changed.
Scrape targets rarely change the set of exposed metrics,
so this optimization should reduce CPU usage in general case.
 2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin
c339642858 lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric 2021-09-11 11:03:38 +03:00
Aliaksandr Valialkin
5aaaa686a4 lib/promscrape: send stale markers for disappeared metrics like Prometheus does 2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin
d613a018c8 lib/promscrape: add ability to set series_limit and stream_parse options via relabeling 
This allows managing these options on a per-target basis.

Typical use case: to manage these options for pods via Kubernetes annotations.
 2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin
b64866e64c lib/promscrape: add scrape_ prefix to job and target labels exported by promscrape_series_limit_rows_dropped_total metric 
This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.
 2021-09-09 17:31:04 +03:00
Aliaksandr Valialkin
c4df601f43 lib/promscrape: add the ability to limit the number of unique series per each scrape target 
The number of series per target can be limited with the following options:

* Global limit with `-promscrape.maxSeriesPerTarget` command-line option.
* Per-target limit with `max_series: N` option in `scrape_config` section.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561
 2021-09-01 16:08:12 +03:00
Aliaksandr Valialkin
146c14d879 lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator 
This is a follow up commit after 31b42b30b6
 2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin
ca61d7c82b lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus 
See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)
 2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin
327034b54f lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices 
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
 2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin
7fdb4db73d lib/promscrape: add ability to load scrape configs from multiple files 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
 2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin
4a2d7aec7f lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config 2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin
67bc407747 lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets 
Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.

This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin
c3b24882a7 lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape 
This will make timestamps for stale markers more consistent for timestamps for other samples
 2021-08-19 14:19:54 +03:00
Aliaksandr Valialkin
8ee575dee9 lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does 2021-08-18 22:00:46 +03:00
Aliaksandr Valialkin
5d92fafc40 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries 
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
 2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin
f21fad53b4 lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag 
This option can be useful when vmagent consumes too much additional memory
for staleness markers functionality and when staleness markers aren't needed.
 2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin
db34c40aec lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets 
This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
 2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin
5f13c519ee lib/promscrape: restore red highlighting for DOWN targets at /targets page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461
 2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin
c1f81f08d4 all: add support for Prometheus staleness markers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin
b877538622 app/vmagent: follow-up after fe445f753b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
 2021-08-05 09:51:00 +03:00
Omar Ghader
fe445f753b
feature: Add multitenant for vmagent (#1505) 
* feature: Add multitenant for vmagent

* Minor fix

* Fix rcs index out of range

* Minor fix

* Fix multi Init

* Fix multi Init

* Fix multi Init

* Add default multi

* Adjust naming

* Add TenantInserted metrics

* Add TenantInserted metrics

* fix: remove unused metrics for vmagent

* fix: remove unused metrics for vmagent

Co-authored-by: mghader <marc.ghader@ubisoft.com>
Co-authored-by: Sebastian YEPES <syepes@gmail.com>
 2021-08-05 09:44:29 +03:00
Aliaksandr Valialkin
77bb9e1656 lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 
See https://github.com/prometheus/prometheus/pull/8978
 2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin
336a2aa2e0 lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does 2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin
c857e05604 lib/promscrape: add missing whitespace at /targets page before up word 2021-07-26 12:23:06 +03:00
Aliaksandr Valialkin
6fc3696260 lib/promscrape/discovery/consul: use case-insensitive comparison for service names 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
 2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin
1c12c0f79c lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does 
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2021-07-01 15:43:43 +03:00
Nikolay
6bd2309449 fixes /targets button style (#1423) 
* fixes /targets button style
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422

* updates boostrap version
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
2edfea8c36 lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID 2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin
0c4c630839 lib/promscrape: typo fix in /targets output 
The typo has been introduced in fb72a2133f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
 2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
4461e20e7d lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 13:22:16 +03:00
Lu Jiajing
12b4cbb68f Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 13:22:16 +03:00
Nikolay
501429c3ff adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin
f10fa0d1d7 lib/promscrape/discovery/consul: properly pass namespace to Consul watcher 
Follow-up for 58a2989fe7
 2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin
4adf6c9766 lib/promscrape/discovery/http: follow up after e307bbb29a 2021-06-22 13:42:10 +03:00
Nikolay
e03a3d3a36 adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin
3ab3902f17 lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does 2021-06-22 13:18:51 +03:00
Nikolay
827a2396d2 adds consul enterprise namespace support (#1400) 
* adds consul enterprise namespace support

* Update lib/promscrape/discovery/consul/consul.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin
f9069ba32a lib/promscrape: show jobs with empty scrape targets on /targets page 2021-06-18 10:54:12 +03:00
Nikolay
9ea1dca3dd fixes DO service discovery labels (#1389) 
adds test for digitalocean sd
 2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin
5f91a701fa lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377
 2021-06-14 14:04:35 +03:00
Nikolay
e42da47608 adds digital ocean sd (#1376) 
* adds digital ocean sd config

* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367

* typo fix
 2021-06-14 13:19:29 +03:00
Aliaksandr Valialkin
df057177a0 lib/promscrape: increase the duration for reading the full response in stream parsing mode 
Increase the duration from 10x to 30x of the configured `scrape_interval'.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin
269e35d676 app/{vmagent,vminsert}: follow-up after 2fe045e2a4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
 2021-06-04 20:33:22 +03:00
jelmd
d8b46908db new feature: debug relabeling (#1344) 
* new feature: relabel logging

Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.

* vminsert wants relabel logging, too.
 2021-06-04 20:33:21 +03:00
Hason Chan
439c2ed510 fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350) 2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin
0b9f0de0a1 lib/promscrape: fix tests after f0c21b6300 2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin
97de72054e docs: document f0c21b6300 2021-05-27 15:04:13 +03:00
faceair
b801b299f0 lib/promscrape: apply body size & sample limit to stream parse (#1331) 
* lib/promscrape: apply body size limit to stream parse

Signed-off-by: faceair <git@faceair.me>

* lib/promscrape: apply sample limit to stream parse

Signed-off-by: faceair <git@faceair.me>
 2021-05-27 15:04:11 +03:00
Nikolay
2780d6dbcd basic OAuth2 support for remoteWrite and scrape targets (#1316) 
* adds OAuth2 support for remoteWrite and scrapping

* adds tests
changes init
 2021-05-22 18:02:01 +03:00