Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-20 15:36:31 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5,777 Commits 114 Branches 748 Tags 619 MiB
5c4f5b83fc
Commit Graph

5777 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aliaksandr Valialkin
c38a10e143
app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks() 2023-01-09 21:08:51 -08:00
Aliaksandr Valialkin
1f9d605988
app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333

This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 ,
but doesn't slow down the common case with merging replicated data blocks so significantly.

Benchmark results:

Before the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13968	     85643 ns/op	 956.53 MB/s	    1700 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	   10806	    109171 ns/op	1500.77 MB/s	    2191 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    8887	    130623 ns/op	1881.45 MB/s	    2660 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    7440	    157348 ns/op	2082.52 MB/s	    3174 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    6534	    184473 ns/op	2220.38 MB/s	    3612 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   13419	     85205 ns/op	 961.44 MB/s	    2213 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     579	   1894900 ns/op	  43.23 MB/s	   46760 B/op	       1 allocs/op

After the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13832	     85298 ns/op	 960.40 MB/s	    1716 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	    8833	    134222 ns/op	1220.66 MB/s	    2675 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    6487	    184830 ns/op	1329.65 MB/s	    3636 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    4977	    236318 ns/op	1386.61 MB/s	    4733 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    4088	    296734 ns/op	1380.36 MB/s	    5761 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   14083	     84067 ns/op	 974.47 MB/s	    2110 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     536	   2043534 ns/op	  40.09 MB/s	   50511 B/op	       1 allocs/op
 2023-01-09 13:01:48 -08:00
Denys Holius
fe0e199859
deployment/docker: update Alertmanager tag from v0.24.0 to v0.25.0 in docker-compose files (#3619) 
deployment/docker: bump alertmanager to latest v0.25.0
 2023-01-09 12:37:14 +01:00
Roman Khavronenko
8aee209c53
lib/promscrape: remove datacenter field from nomad_sd_config (#3612) 
Looks like `datacenter` field isn't part of `/v1/services` API.
See https://developer.hashicorp.com/nomad/api-docs/services#list-services
and https://developer.hashicorp.com/nomad/api-docs/services#read-service

Related issues:
https://github.com/traefik/traefik/issues/9109
https://github.com/prometheus/prometheus/issues/11776

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-09 09:07:40 +01:00
Aliaksandr Valialkin
28f8dc41b0
lib/promscrape/discoveryutils: cleanup after 5df9fddaf2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-07 01:26:54 -08:00
Zakhar Bessarab
5df9fddaf2
lib/promscrape/discoveryutils: use correct timeout for blocking requests (#3609) 
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-07 01:13:03 -08:00
Aliaksandr Valialkin
41e00a0df7
lib/storage: simplify the fix from 488940502c 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3566
 2023-01-07 01:04:43 -08:00
Dmytro Kozlov
488940502c
lib/storage: fix returning camelcase label names (#3608) 
* lib/storage: fix returning camelcase label names

* doc: add change log

* Update docs/CHANGELOG.md

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-07 00:50:14 -08:00
Aliaksandr Valialkin
5fe7ff24c2
lib/streamaggr: limit the the number of concurrent flushes of the aggregate data to the exact number of available CPUs 
This should reduce the maximum memory usage during concurrent flushes of the aggregate data
 2023-01-07 00:18:51 -08:00
Aliaksandr Valialkin
ad5bfe3089
lib/promscrape: reduce the number of concurrently executed processScrapedData calls from 2x of the number of CPUs to the number of CPUs 
This should reduce the maximum memory usage for processScrapedData() function by 2x.
The only part, which can be IO-bound in the processScrapedData() is pushData() call,
when it buffers data to persistent queue if the remote storage cannot keep up
with the data ingestion speed. In this case it is OK if the scrape pace will be limited.
 2023-01-07 00:14:30 -08:00
Aliaksandr Valialkin
af263fe881
all: small improvements in error messages and command-line flag descriptions related to concurrency limiters 2023-01-07 00:11:44 -08:00
Aliaksandr Valialkin
45f39e291e
lib/writeconcurrencylimiter: moved the error generation from incConcurrency() to the caller place 2023-01-06 23:45:58 -08:00
Aliaksandr Valialkin
986a05e18d
lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples 
This should reduce memory usage when scraping big number of targets,
since this limits the summary memory usage during concurrent parsing and relabeling
by the number of available CPU cores.
 2023-01-06 22:59:17 -08:00
Aliaksandr Valialkin
293e4dc77b
app/{vminsert,vmstorage}: add comments on why storage.AddRows() is called without limiting the number of concurrent calls 2023-01-06 22:40:07 -08:00
Aliaksandr Valialkin
5c4bd4f7c1
lib/streamaggr: limit the number of concurrent flushes of aggregate metrics in order to limit memory usage 2023-01-06 22:39:13 -08:00
Aliaksandr Valialkin
c63755c316
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit 
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
 2023-01-06 22:20:19 -08:00
Aliaksandr Valialkin
f299d2ca1a
lib/vmselectapi: limit the number of concurrently executed requests 
This should prevent from out of memory errors when big number of vmselect
nodes send many concurrent requests to vmstorage

The limit can be controlled at vmstorage via the following command-line flags:
- search.maxConcurrentRequests
- search.maxQueueDuration

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits
 2023-01-06 22:11:34 -08:00
Aliaksandr Valialkin
e7637885a6
app/vmselect: improve error message when the request cannot be started because too many concurrent requests are already executed 2023-01-06 22:10:42 -08:00
Aliaksandr Valialkin
463b957e54
lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 21:52:33 -08:00
Aliaksandr Valialkin
f392913d00
lib/promscrape: follow-up after bced9fb978 
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-05 21:13:06 -08:00
Zakhar Bessarab
bced9fb978
lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568) 2023-01-05 19:34:47 -08:00
Roman Khavronenko
5bdd880142
vmstorage: add more context to the flock acquiring msg (#3584) 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3578

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-05 18:30:42 -08:00
Aliaksandr Valialkin
9f348cf8a1
lib/promscrape/discovery/nomad: follow-up after 48f371a46c 
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 18:07:20 -08:00
Aliaksandr Valialkin
cad8553c01
Makefile: remove trailing space after golangci-lint run command 
It is left after ec2c82e800
 2023-01-05 16:59:07 -08:00
Aliaksandr Valialkin
1a28f0e5b3
lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 
This allows copy-n-pasting the url to another browser window and seeing the same result.

The limit in 1000 chars is selected in order to prevent from potential issues with systems
which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string

If the limit is exceeded, then query args are sent via POST method and aren't visible in the url.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 16:48:04 -08:00
Karan Sharma
48f371a46c
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549) 
Add nomad_sd_config support for service discovery
 2023-01-05 23:03:58 +01:00
Denys Holius
043b28c725
.github/workflows/nightly-build.yml: added dockerhub login (#3594) 2023-01-05 16:54:14 +01:00
Luke Palmer
ec2c82e800
Lint and errcheck using golangci-lint (#3558) 2023-01-05 16:12:46 +01:00
Zakhar Bessarab
01bc0c94ab
doc: add vmbackupmanager monitoring section (#3605) 
* doc: add vmbackupmanager monitoring section

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 16:03:06 +01:00
Thomas Danielsson
9d1104d812
dashboards: fix operator datasource variable (#3604) 
Got "Failed to upgrade legacy queries Datasource $ds was not found" in
Grafana on operator dashboard.
It's datasource variable was incorrectly named `datasource`.

Also made the rest of the dashboards have homogeneous datasource-variable
names and selections, matching vmagent dashboard.
 2023-01-05 14:59:56 +01:00
Artem Navoiev
8b763175ff
Add Understand Your Setup Size Guide (#3572) 
docs: add Understand Your Setup Size Guide

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
 2023-01-05 14:56:50 +01:00
Aliaksandr Valialkin
2ee81a5dbb
docs/CHANGELOG.md: add missing dot 2023-01-05 03:35:02 -08:00
Zakhar Bessarab
185cdcd813
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586) 
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 03:34:25 -08:00
Aliaksandr Valialkin
0dea3b71da
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2023-01-05 03:26:49 -08:00
Aliaksandr Valialkin
a1076abcbf
lib/promscrape: follow-up for a7e29c38bc 
- Document the bugfix at docs/CHANGELOG.md
- Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 02:52:08 -08:00
Zakhar Bessarab
a7e29c38bc
lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595) 
* lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/targetstatus: address review comment

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 02:39:31 -08:00
Yury Molodov
2460e0f51e
vmui: improve Explore metrics (#3598) 
* feat: add multiple select

* feat: improve explore interface

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 02:23:04 -08:00
Aliaksandr Valialkin
0e1f0ade31
lib/streamaggr: sort by and without labels in the aggregate output metric name 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
 2023-01-05 02:08:44 -08:00
Aliaksandr Valialkin
04dff34de4
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0 
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
 2023-01-05 01:50:10 -08:00
Aliaksandr Valialkin
66947ee5a2
lib/streamaggr: remove unused fields 2023-01-04 13:33:46 -08:00
Roman Khavronenko
9d0e1f8e68
dashboards: add backupmanager dashboard (#3599) 
Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-04 17:26:15 +01:00
Roman Khavronenko
63bf583b3c
github: rm plaintext render (#3597) 
`render: plain text` makes fields not formattable and prevents
 from pasting screenshots.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-04 14:48:17 +01:00
Denys Holius
09fe346d18
docs/Release-Guide.md: adds release scenario for RPM LTS packages (#3588) 2023-01-04 14:36:14 +01:00
Zakhar Bessarab
59f20c1034
github: use github templates for filling in feature requests or bug reports (#3587) 
github: use github templates for filling in feature requests or bug reports
 2023-01-04 14:34:19 +01:00
Artem Navoiev
0ff85d00a4
update year in License 
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
 2023-01-04 09:46:38 +01:00
Aliaksandr Valialkin
fafece1af8
vendor: make vendor-update 2023-01-03 23:36:42 -08:00
Aliaksandr Valialkin
5bca3a5be2
app/vmselect: remove dependency on lib/promscrape from app/vmselect 2023-01-03 23:28:27 -08:00
Aliaksandr Valialkin
fd175ad80b
docs: update -help outputs for vm* tools 2023-01-03 23:27:06 -08:00
Aliaksandr Valialkin
fa13bbc48a
app/{vmagent,vminsert}: add support for streaming aggregation 
See https://docs.victoriametrics.com/stream-aggregation.html

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
 2023-01-03 22:19:21 -08:00
Aliaksandr Valialkin
add2c4bf07
lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..)) 2023-01-03 22:16:22 -08:00