Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 08:23:34 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5,512 Commits 114 Branches 748 Tags 504 MiB
96f04c9863
Commit Graph

5512 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aliaksandr Valialkin
96f04c9863
app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333

This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 ,
but doesn't slow down the common case with merging replicated data blocks so significantly.

Benchmark results:

Before the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13968	     85643 ns/op	 956.53 MB/s	    1700 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	   10806	    109171 ns/op	1500.77 MB/s	    2191 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    8887	    130623 ns/op	1881.45 MB/s	    2660 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    7440	    157348 ns/op	2082.52 MB/s	    3174 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    6534	    184473 ns/op	2220.38 MB/s	    3612 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   13419	     85205 ns/op	 961.44 MB/s	    2213 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     579	   1894900 ns/op	  43.23 MB/s	   46760 B/op	       1 allocs/op

After the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13832	     85298 ns/op	 960.40 MB/s	    1716 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	    8833	    134222 ns/op	1220.66 MB/s	    2675 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    6487	    184830 ns/op	1329.65 MB/s	    3636 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    4977	    236318 ns/op	1386.61 MB/s	    4733 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    4088	    296734 ns/op	1380.36 MB/s	    5761 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   14083	     84067 ns/op	 974.47 MB/s	    2110 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     536	   2043534 ns/op	  40.09 MB/s	   50511 B/op	       1 allocs/op
 2023-01-09 12:58:18 -08:00
Aliaksandr Valialkin
7792ba3272
lib/promscrape/discoveryutils: cleanup after 5df9fddaf2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-07 01:27:16 -08:00
Zakhar Bessarab
e8624fd781
lib/promscrape/discoveryutils: use correct timeout for blocking requests (#3609) 
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-07 01:27:10 -08:00
Aliaksandr Valialkin
eb9a542c1f
lib/storage: simplify the fix from 488940502c 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3566
 2023-01-07 01:11:35 -08:00
Dmytro Kozlov
f739e44802
lib/storage: fix returning camelcase label names (#3608) 
* lib/storage: fix returning camelcase label names

* doc: add change log

* Update docs/CHANGELOG.md

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-07 01:11:10 -08:00
Aliaksandr Valialkin
c630115be0
lib/streamaggr: limit the the number of concurrent flushes of the aggregate data to the exact number of available CPUs 
This should reduce the maximum memory usage during concurrent flushes of the aggregate data
 2023-01-07 00:19:34 -08:00
Aliaksandr Valialkin
0a14b7bb82
lib/promscrape: reduce the number of concurrently executed processScrapedData calls from 2x of the number of CPUs to the number of CPUs 
This should reduce the maximum memory usage for processScrapedData() function by 2x.
The only part, which can be IO-bound in the processScrapedData() is pushData() call,
when it buffers data to persistent queue if the remote storage cannot keep up
with the data ingestion speed. In this case it is OK if the scrape pace will be limited.
 2023-01-07 00:17:52 -08:00
Aliaksandr Valialkin
5876821a16
all: small improvements in error messages and command-line flag descriptions related to concurrency limiters 2023-01-07 00:12:24 -08:00
Aliaksandr Valialkin
3864357772
lib/writeconcurrencylimiter: moved the error generation from incConcurrency() to the caller place 2023-01-07 00:01:44 -08:00
Aliaksandr Valialkin
7fb02f536a
lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples 
This should reduce memory usage when scraping big number of targets,
since this limits the summary memory usage during concurrent parsing and relabeling
by the number of available CPU cores.
 2023-01-06 23:01:18 -08:00
Aliaksandr Valialkin
3461ae8f13
lib/streamaggr: limit the number of concurrent flushes of aggregate metrics in order to limit memory usage 2023-01-06 22:40:19 -08:00
Aliaksandr Valialkin
2ca48444e2
lib/vmselectapi: typo fix after 20e9598254 2023-01-06 22:13:32 -08:00
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit 
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
 2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin
20e9598254
lib/vmselectapi: limit the number of concurrently executed requests 
This should prevent from out of memory errors when big number of vmselect
nodes send many concurrent requests to vmstorage

The limit can be controlled at vmstorage via the following command-line flags:
- search.maxConcurrentRequests
- search.maxQueueDuration

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits
 2023-01-06 18:39:46 -08:00
Aliaksandr Valialkin
cd705b0f69
app/vmselect: improve error message when the request cannot be started because too many concurrent requests are already executed 2023-01-06 18:19:05 -08:00
Aliaksandr Valialkin
be896ddfd4
lib/protoparser/clusternative: typo fix in the comment: thic -> this 2023-01-06 18:16:25 -08:00
Aliaksandr Valialkin
ec7a3b79ab
lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 21:53:08 -08:00
Aliaksandr Valialkin
54410bf51b
lib/promscrape: follow-up after bced9fb978 
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-05 21:23:21 -08:00
Zakhar Bessarab
de5aad2cde
lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568) 2023-01-05 21:23:15 -08:00
Roman Khavronenko
57277ed6bc
vmstorage: add more context to the flock acquiring msg (#3584) 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3578

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-05 18:32:53 -08:00
Aliaksandr Valialkin
750d309f63
lib/promscrape/discovery/nomad: follow-up after 48f371a46c 
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 18:09:23 -08:00
Aliaksandr Valialkin
87a4943bea
Makefile: remove trailing space after golangci-lint run command 
It is left after ec2c82e800
 2023-01-05 18:09:05 -08:00
Karan Sharma
8f42c5a024
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549) 
Add nomad_sd_config support for service discovery
 2023-01-05 18:07:02 -08:00
Luke Palmer
51d956225d
Lint and errcheck using golangci-lint (#3558) 2023-01-05 17:00:17 -08:00
Zakhar Bessarab
eee7e75278
doc: add vmbackupmanager monitoring section (#3605) 
* doc: add vmbackupmanager monitoring section

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 16:49:59 -08:00
Thomas Danielsson
ec1f6811a1
dashboards: fix operator datasource variable (#3604) 
Got "Failed to upgrade legacy queries Datasource $ds was not found" in
Grafana on operator dashboard.
It's datasource variable was incorrectly named `datasource`.

Also made the rest of the dashboards have homogeneous datasource-variable
names and selections, matching vmagent dashboard.
 2023-01-05 16:49:19 -08:00
Artem Navoiev
9d85c62ce2
Add Understand Your Setup Size Guide (#3572) 
docs: add Understand Your Setup Size Guide

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
 2023-01-05 16:48:28 -08:00
Aliaksandr Valialkin
6eda4c6da2
lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum 
In this case there is no need in converting int64 to int
 2023-01-05 16:46:03 -08:00
Aliaksandr Valialkin
1af6e0b233
lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 
This allows copy-n-pasting the url to another browser window and seeing the same result.

The limit in 1000 chars is selected in order to prevent from potential issues with systems
which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string

If the limit is exceeded, then query args are sent via POST method and aren't visible in the url.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 16:45:42 -08:00
Aliaksandr Valialkin
21be927afa
docs/CHANGELOG.md: add missing dot 2023-01-05 03:35:20 -08:00
Zakhar Bessarab
99f9b02283
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586) 
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin
1d16cc9349
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin
634e24e685
lib/promscrape: follow-up for a7e29c38bc 
- Document the bugfix at docs/CHANGELOG.md
- Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 02:51:45 -08:00
Zakhar Bessarab
52226c392f
lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595) 
* lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/targetstatus: address review comment

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 02:48:39 -08:00
Yury Molodov
802febab74
vmui: improve Explore metrics (#3598) 
* feat: add multiple select

* feat: improve explore interface

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 02:24:05 -08:00
Aliaksandr Valialkin
c97d6ed6a4
lib/streamaggr: sort by and without labels in the aggregate output metric name 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
 2023-01-05 02:08:59 -08:00
Aliaksandr Valialkin
78114e85d6
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0 
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
 2023-01-05 01:50:26 -08:00
Roman Khavronenko
7c79a6f4ac
dashboards: add backupmanager dashboard (#3599) 
Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-04 13:34:04 -08:00
Aliaksandr Valialkin
ccec8c26ed
lib/streamaggr: remove unused fields 2023-01-04 13:33:21 -08:00
Roman Khavronenko
23473de81e
github: rm plaintext render (#3597) 
`render: plain text` makes fields not formattable and prevents
 from pasting screenshots.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-04 08:44:32 -08:00
Denys Holius
28649f0773
docs/Release-Guide.md: adds release scenario for RPM LTS packages (#3588) 2023-01-04 08:39:10 -08:00
Zakhar Bessarab
e38100fbb5
github: use github templates for filling in feature requests or bug reports (#3587) 
github: use github templates for filling in feature requests or bug reports
 2023-01-04 08:27:59 -08:00
Artem Navoiev
49f30320d6
update year in License 
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
 2023-01-04 08:20:14 -08:00
Aliaksandr Valialkin
0c7d161858
vendor: make vendor-update 2023-01-03 23:36:36 -08:00
Aliaksandr Valialkin
ebb8aeb0cf
app/vmselect: remove dependency on lib/promscrape from app/vmselect 2023-01-03 23:27:36 -08:00
Aliaksandr Valialkin
ac890b3081
docs: update -help outputs for vm* tools 2023-01-03 23:27:31 -08:00
Aliaksandr Valialkin
3369371636
app/{vmagent,vminsert}: add support for streaming aggregation 
See https://docs.victoriametrics.com/stream-aggregation.html

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
 2023-01-03 22:22:07 -08:00
Aliaksandr Valialkin
d3f8298739
lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..)) 2023-01-03 22:15:49 -08:00
Aliaksandr Valialkin
cfba42091e
app/vmagent/remotewrite: improve descriptions for -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig 
- Cross-reference these command-line flags.
- Add a link to https://docs.victoriametrics.com/vmagent.html#relabeling
 2023-01-03 22:05:39 -08:00
Aliaksandr Valialkin
b49927bcb1
docs/Single-server-VictoriaMetrics.md: impromve formatting for prominent features chapter 2023-01-03 21:58:38 -08:00