Aliaksandr Valialkin
114cf24b43
lib/promscrape/discovery/dns: add __meta_dns_srv_record_target
and __meta_dns_srv_record_port
labels
...
This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases
and https://github.com/prometheus/prometheus/pull/7678 .
2020-09-11 21:35:39 +03:00
Aliaksandr Valialkin
5cf5a0e8c4
lib/protoparser/common: do not read request body when parsing timestamp
query arg
...
This was preventing from reading data via /api/v1/prometheus/import .
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin
81c05f669b
lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin
8bd5aa3516
lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus
...
This improves consistency with timestamps in Prometheus text exposition format
2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin
58d3b82ae5
app/{vminsert,vmagent}: allow passing timestamp via timestamp
query arg when ingesting data to /api/v1/import/prometheus
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
2020-09-11 13:28:31 +03:00
Nikolay Khramchikhin
af994562c8
Added endpointslices discovery to k8s api ( #760 )
...
This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21.
See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1
* Added endpointslices discovery to k8s api
Started from 1.17 k8s version endpointslices is beta,
it allows to query k8s api for endpoints more efficient.
It presents at scrape_config.yaml as separate role for kubernetes_sd_config.
kubernetes_sd_config:
- role: endpointslices
* fixed typos, changed EndpointConditions signature - with values instead of pointers
2020-09-11 12:24:50 +03:00
Aliaksandr Valialkin
f307e6f432
app/vmselect: initial implementation of Graphite Metrics API
...
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
f5cb213ef9
lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps
...
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.
Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin
e5c8377212
lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem
2020-09-08 19:57:20 +03:00
Nikolay Khramchikhin
fb356c434b
Changed s3 configProfile flag default, ( #749 )
...
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin
d2f6f96e4a
lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699
2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin
281d715060
lib/httpserver: add -http.connTimeout
command-line flag for limiting the lifetime for incoming http connections
...
This can be useful for balancing incoming connections among multiple services.
2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin
4fa97430d7
app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats
...
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
95ce89e7d7
lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling
...
The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc
,
because `sw.labels` may be reset multiple times when processing big number of rows.
2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin
bc1ca4b20b
lib/httpserver: add -http.idleConnTimeout
command-line flag for tuning the timeout for incoming idle http connections
2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin
a01c56104a
lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics
...
This has been broken at 7785869ccc
2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin
deff8d419a
lib/promscrape: reduce memory usage when scraping targets with millions of metrics
...
This should help when scraping /federate endpoints from Prometheus instances,
which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/
2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin
5f2277624a
lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy
2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin
45e770ed20
lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores
...
There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.
2020-08-28 09:49:22 +03:00
Roman Khavronenko
34ef10fbcc
lib/flagutil: avoid int overflow for arch 386 ( #710 )
...
Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32,
whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with
`int overflow` error.
2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin
9a77ae9d1c
lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels
...
Previously targets with big number of metrics and/or labels could generated too big buffers,
which then could be re-used when scraping targets with small number of metrics.
This resulted in memory waste.
Now big buffers are used only for targets with big number of metrics / labels,
while small buffers are used for targets with small number of metrics / labels.
2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin
3ea6444219
lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket
...
This should reduce the number of resizings for the returned byte buffers.
2020-08-16 22:13:38 +03:00
Roman Khavronenko
4b89da9463
lib/decimal: rename significant decimal digits
to significant figures
( #698 )
...
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures ) rounding
applied to all significant figures, not just decimal ones.
2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989
all: allow using KB
, MB
, GB
, KiB
, MiB
and GiB
suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
1b5467f7fd
lib/memory: improve log message about the memory allowed to use by VictoriaMetrics
2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin
d9f7ea1c6e
lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data
...
The OS should return any buffered data in the stream without the need to set the read timeout.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin
edf3ee2a9b
lib: dump compressed block contents on error during decompression
...
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
6e863376f7
lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty
...
This should reduce memory allocations and copying when the byte slice is growing.
2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin
3efa4e4e1c
lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin
c6b0547847
lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin
9d79a3a99d
lib/memory: add -memory.allowedBytes
command-line flag for setting absolute memory limit for VictoriaMetrics caches
2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin
b996280c65
app/{vminsert,vmagent}: improve documentation for -influxListenAddr
command-line flag
2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
c82a485cf6
lib/protoparser/prometheus: typo fix in error message
2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin
9e67343756
lib/promscrape: use a hint on body length instead of body capacity
...
This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin
b4119bb51e
lib/promscrape: reduce memory usage when scraping big number of targets
...
Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin
1724cc241e
lib/promscrape: properly retry requests on the server closed connection before returning the first response byte
error during service discover API calls and target scrapes
2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin
60c7397be5
all: support %{ENV_VAR}
placeholders in yaml configs in all the vm* components
...
Such placeholders are substituted by the corresponding environment variable values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9
app: respect CPU limits set via cgroups
...
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
59d95961b8
lib/protoparser: clarify that the string passed to Unmarshal()
function must remain available when the parsed rows are in use
2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin
890cfe5b61
lib/protoparser/influx: accept precision=us
and precision=µ
according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint
2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin
e3439a6cd0
lib/promscrape: optimize per-metric hash calculations
...
This increases vmagent performance by up to 10% when scraping big number of metrics
2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin
4ce1368e4b
lib/storage: mention time range used in the query that led to error message
...
This should improve detecting slow queries with too big time ranges
2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin
f92255e803
lib/storage: mention tag filters used in the query that led to error message
...
This should improve detecting invalid or heavy queries that lead to errors.
2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin
b3d4ff7ee2
app/vmstorage: improve error logging when the request times out
2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin
e3999ac010
lib/promscrape: show real timestamp and real duration for the scape on /targets
page
...
Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp
2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin
c09c881264
lib/promscrape: make errcheck happy
2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin
2dfb42a8b4
lib/promscrape: export scrape_samples_added
per-target metric like Prometheus does
...
This metric may be useful for detecting targets with high churn rate for the exported metrics.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683
2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin
fd9f1463df
lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails
...
this is expected condition, so it is better to use WARN log level for it
2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin
d4be3efc60
lib/promscrape: add a test for scrape config for blackbox exporter
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684
2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin
67cacb22ac
lib/httpserver: add -tls
, -tlsCertFile
and -tlsKeyFile
command-line flags in every vm binary
...
This makes such binaries compatible with binaries from `master` branch (aka single-node version)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
2020-08-07 10:57:32 +03:00