Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 00:13:30 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,550 Commits 114 Branches 748 Tags 504 MiB
7ab7ae79c7
Commit Graph

586 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
4fa97430d7 app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
95ce89e7d7 lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling 
The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc ,
because `sw.labels` may be reset multiple times when processing big number of rows.
 2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin
bc1ca4b20b lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections 2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin
a01c56104a lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics 
This has been broken at 7785869ccc
 2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin
deff8d419a lib/promscrape: reduce memory usage when scraping targets with millions of metrics 
This should help when scraping /federate endpoints from Prometheus instances,
which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/
 2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin
5f2277624a lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy 2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin
45e770ed20 lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores 
There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.
 2020-08-28 09:49:22 +03:00
Roman Khavronenko
34ef10fbcc lib/flagutil: avoid int overflow for arch 386 (#710) 
Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32,
whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with
`int overflow` error.
 2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin
9a77ae9d1c lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels 
Previously targets with big number of metrics and/or labels could generated too big buffers,
which then could be re-used when scraping targets with small number of metrics.
This resulted in memory waste.

Now big buffers are used only for targets with big number of metrics / labels,
while small buffers are used for targets with small number of metrics / labels.
 2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin
3ea6444219 lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket 
This should reduce the number of resizings for the returned byte buffers.
 2020-08-16 22:13:38 +03:00
Roman Khavronenko
4b89da9463 lib/decimal: rename significant decimal digits to significant figures (#698) 
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding
applied to all significant figures, not just decimal ones.
 2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
1b5467f7fd lib/memory: improve log message about the memory allowed to use by VictoriaMetrics 2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin
d9f7ea1c6e lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data 
The OS should return any buffered data in the stream without the need to set the read timeout.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin
edf3ee2a9b lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
6e863376f7 lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty 
This should reduce memory allocations and copying when the byte slice is growing.
 2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin
3efa4e4e1c lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin
c6b0547847 lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin
9d79a3a99d lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches 2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin
b996280c65 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
c82a485cf6 lib/protoparser/prometheus: typo fix in error message 2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin
9e67343756 lib/promscrape: use a hint on body length instead of body capacity 
This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
 2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin
b4119bb51e lib/promscrape: reduce memory usage when scraping big number of targets 
Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
 2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin
1724cc241e lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes 2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin
60c7397be5 all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 
Such placeholders are substituted by the corresponding environment variable values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
 2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
59d95961b8 lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use 2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin
890cfe5b61 lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint 2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin
e3439a6cd0 lib/promscrape: optimize per-metric hash calculations 
This increases vmagent performance by up to 10% when scraping big number of metrics
 2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin
4ce1368e4b lib/storage: mention time range used in the query that led to error message 
This should improve detecting slow queries with too big time ranges
 2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin
f92255e803 lib/storage: mention tag filters used in the query that led to error message 
This should improve detecting invalid or heavy queries that lead to errors.
 2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin
b3d4ff7ee2 app/vmstorage: improve error logging when the request times out 2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin
e3999ac010 lib/promscrape: show real timestamp and real duration for the scape on /targets page 
Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp
 2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin
c09c881264 lib/promscrape: make errcheck happy 2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin
2dfb42a8b4 lib/promscrape: export scrape_samples_added per-target metric like Prometheus does 
This metric may be useful for detecting targets with high churn rate for the exported metrics.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683
 2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin
fd9f1463df lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails 
this is expected condition, so it is better to use WARN log level for it
 2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin
d4be3efc60 lib/promscrape: add a test for scrape config for blackbox exporter 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684
 2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin
67cacb22ac lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary 
This makes such binaries compatible with binaries from `master` branch (aka single-node version)

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
 2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin
307281e922 lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit 
This should improve data ingestion performance when heavy searches are executed

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin
dd1d59f57a lib/storage: properly check timeouts and pace limits 
Previously they were checked on every iteration for small number of iterations
 2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin
a2039b3bbc app/vmselect: return the upper bound on the number of found time series from storage.Search.Init 
This is used by a single-node version in order to reduce memory allocations during search.
See bc8381613d for details.
 2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin
b690eeff53 lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x 2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin
6c0a92a1ee lib/pacelimiter: increase scalability for multi-CPU system 2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin
13f8644f8e lib/storage: optimize prefetching metric names for the given metricIDs 2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin
f789e0fa44 lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock 2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin
a3e91c593b lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
76064ba9e7 Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer 2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin
8cc2e01386 lib/backup: allow using ~/.aws/config without region 
Use us-west-2 for determining bucket region.
 2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin
a2aa3a60eb app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
3149af624d lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 
Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this
increases query latencies since more CPUs are busy with merges.

While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.
 2020-07-31 17:53:13 +03:00