Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page

Browse Source 
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
This commit is contained in:
Aliaksandr Valialkin 2020-04-14 14:51:52 +03:00
parent 391fb0903e
commit 6ec582acb9
3 changed files with 15 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
app/vmagent/README.md
View File

@ -183,7 +183,7 @@ Read more about relabeling in the following articles:
`vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page `vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page
either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later. either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later.
`vmagent` also exports target statuses at `http://vmagent-host:8429/targets` page in plaintext format. `vmagent` also exports target statuses at `http://vmagent-host:8429/targets` page in plaintext format. This page also exports information on improperly configured scrape configs.
### Troubleshooting ### Troubleshooting

2
docs/vmagent.md
View File

@ -183,7 +183,7 @@ Read more about relabeling in the following articles:
`vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page `vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. It is recommended setting up regular scraping of this page
either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later. either via `vmagent` itself or via Prometheus, so the exported metrics could be analyzed later.
`vmagent` also exports target statuses at `http://vmagent-host:8429/targets` page in plaintext format. `vmagent` also exports target statuses at `http://vmagent-host:8429/targets` page in plaintext format. This page also exports information on improperly configured scrape configs.
### Troubleshooting ### Troubleshooting

13
lib/promscrape/targetstatus.go
View File

@ -86,6 +86,7 @@ func (tsm *targetStatusMap) WriteHumanReadable(w io.Writer) {
return jss[i].job < jss[j].job return jss[i].job < jss[j].job
}) })
targetsByEndpoint := make(map[string]int)
for _, js := range jss { for _, js := range jss {
sts := js.statuses sts := js.statuses
sort.Slice(sts, func(i, j int) bool { sort.Slice(sts, func(i, j int) bool {
@ -115,8 +116,20 @@ func (tsm *targetStatusMap) WriteHumanReadable(w io.Writer) {
} }
fmt.Fprintf(w, "\tstate=%s, endpoint=%s, labels=%s, last_scrape=%.3fs ago, scrape_duration=%.3fs, error=%q\n", fmt.Fprintf(w, "\tstate=%s, endpoint=%s, labels=%s, last_scrape=%.3fs ago, scrape_duration=%.3fs, error=%q\n",
state, st.sw.ScrapeURL, labelsStr, lastScrape.Seconds(), float64(st.scrapeDuration)/1000, errMsg) state, st.sw.ScrapeURL, labelsStr, lastScrape.Seconds(), float64(st.scrapeDuration)/1000, errMsg)
key := fmt.Sprintf("endpoint=%s, labels=%s", st.sw.ScrapeURL, labelsStr)
targetsByEndpoint[key]++
} }
} }
fmt.Fprintf(w, "\n")
// Check whether there are targets with duplicate endpoints and labels.
for key, n := range targetsByEndpoint {
if n <= 1 {
continue
}
fmt.Fprintf(w, "!!! Scrape config error: %d duplicate targets with identical endpoint and labels found:\n", n)
fmt.Fprintf(w, "\t%s\n", key)
}
} }
type jobStatus struct { type jobStatus struct {