Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-15 16:30:55 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,981 Commits 114 Branches 748 Tags 504 MiB
bdac2171f1
Commit Graph

741 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
097a4c10dd app/vmstorage: add metrics for determining whether background merges need additional disk space to complete 
These metrics are:

* vm_small_merge_need_free_disk_space
* vm_big_merge_need_free_disk_space

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin
5dca7bbe85 app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin
5e998e597a lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online 
The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users
as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309
 2020-09-29 13:55:53 +03:00
Aliaksandr Valialkin
338a53ccf9 lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm 2020-09-29 13:10:37 +03:00
Aliaksandr Valialkin
ef416c72c2 lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm 2020-09-29 12:42:25 +03:00
Aliaksandr Valialkin
0b0259c42c lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic 2020-09-29 12:29:50 +03:00
Aliaksandr Valialkin
8f25206d8c lib/protoparser/prometheus: fix TestParseStream after 124f78857b 2020-09-29 12:12:33 +03:00
Aliaksandr Valialkin
81cdf2fa14 lib/{fs,filestream}: small consistency-related updates after cc90a548b1 2020-09-29 00:43:20 +03:00
Nikolay Khramchikhin
658a05ef0f added openbsd implementations (#790) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785

removed fadvise for openbsd, added freespace implemenation for openbsd
 2020-09-29 00:43:19 +03:00
Aliaksandr Valialkin
1481d6d8ff lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer 2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin
8df33bd5c1 app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
7072db75cb lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server 
This should reduce syscall overhead when reading big amounts of data
 2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin
6d8c23fdbd app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native 
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
 2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
00ec2b7189 lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream 
Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.
 2020-09-26 04:22:06 +03:00
Aliaksandr Valialkin
533bf76a12 lib/storage: correctly use maxBlockSize in various checks 
Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.
 2020-09-24 18:13:15 +03:00
Aliaksandr Valialkin
543f3aea97 all: consistently use "%w" formatting in fmt.Errorf for wrapped errors 2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin
90d2549428 lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue 2020-09-23 02:17:53 +03:00
Aliaksandr Valialkin
89d652b583 lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728
 2020-09-22 23:27:26 +03:00
Aliaksandr Valialkin
31e341371b lib/storage: code prettifying after be5e1222f3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
 2020-09-22 00:42:20 +03:00
faceair
ad41e39350 add filter to getMetricIDs (#783) 
* add getMetricIDs filter

* check nil filter before use
 2020-09-22 00:42:19 +03:00
Aliaksandr Valialkin
d32c3f747c lib/logger: add -loggerDisableTimestamps command-line flag for disabling timestamps in logs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778
 2020-09-21 19:25:50 +03:00
Aliaksandr Valialkin
f961838290 lib/promscrape/discovery/ec2: code prettifying after 312fead9a2 2020-09-21 18:44:04 +03:00
Nikolay Khramchikhin
0069353d5e Add improvements to ec2_sd_discovery (#775) 
* Add improvements to ec2 discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771

 role_arn support with aws sts
 instance iam_role support
 refreshing temporary tokens

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* changed implementation, removed tests, clean up code

* moved endpoint builder into getEC2APIResponse

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin
a9321f6a60 lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges 2020-09-21 15:51:26 +03:00
Aliaksandr Valialkin
604e8f6114 lib/decimal: optimize maxUpExponent() by eliminating division from hot path 2020-09-19 13:49:43 +03:00
Aliaksandr Valialkin
df547bf345 lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush 2020-09-19 12:51:46 +03:00
Aliaksandr Valialkin
778ea183ca lib/decimal: properly store Inf values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-18 19:08:53 +03:00
Aliaksandr Valialkin
adb4d3b75c lib/persistentqueue: flush data to disk every second 
Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
 2020-09-18 13:06:03 +03:00
Aliaksandr Valialkin
8fd791d399 lib/promscrape: avoid copying response body when scraping targets. 
This should reduce memory usage when scraping targets with millions of metrics.
 2020-09-18 13:06:00 +03:00
Aliaksandr Valialkin
d96858b921 lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions 
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin
3abbb38254 lib/{mergeset,storage}: compare errors with errors.Is() 2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin
ddb3519e17 lib/{mergeset,storage}: code prettifying 2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin
bf826dd828 lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions 2020-09-17 01:49:08 +03:00
Aliaksandr Valialkin
406f4fe445 app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens 
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin
9705ac5d7a lib/persistentqueue: code simplification after d455764a6f 2020-09-16 21:14:01 +03:00
Aliaksandr Valialkin
eee6f1e56d lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset) 
The strategy is:

- Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown.
- Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data.
- Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown.
- Skipping of broken chunk file if it cannot be read.
- Fsyncing finalized chunk files.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
 2020-09-16 18:13:24 +03:00
Aliaksandr Valialkin
6ce52e3702 lib/protoparser/vmimport: add more testcases for invalid timestamps and values 
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
 2020-09-16 02:21:53 +03:00
Aliaksandr Valialkin
cd87ca303f lib/protoparser: report more errors for incorrect timestamps and/or values 
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.

Updates https://github.com/VictoriaMetrics/vmctl/issues/25
 2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin
5c4e111b43 lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
 2020-09-16 02:16:13 +03:00
Aliaksandr Valialkin
0c1c1b79ba lib/promscrape: add a link to troubleshooting docs to error message when duplicate scrape target with identical labels is skipped 2020-09-15 14:16:20 +03:00
Aliaksandr Valialkin
ca08161b54 lib/promscrape: typo fix 2020-09-12 00:14:15 +03:00
Aliaksandr Valialkin
e53235ac5c lib/promscrape: do not reset the remaining rows when pushing a part of data to remote storage during big scrapes 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753

Thanks to @PerGon and @clmssz for help with debugging.
 2020-09-11 23:38:17 +03:00
Aliaksandr Valialkin
114cf24b43 lib/promscrape/discovery/dns: add __meta_dns_srv_record_target and __meta_dns_srv_record_port labels 
This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases
and https://github.com/prometheus/prometheus/pull/7678 .
 2020-09-11 21:35:39 +03:00
Aliaksandr Valialkin
5cf5a0e8c4 lib/protoparser/common: do not read request body when parsing timestamp query arg 
This was preventing from reading data via /api/v1/prometheus/import .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin
81c05f669b lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
 2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin
8bd5aa3516 lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus 
This improves consistency with timestamps in Prometheus text exposition format
 2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin
58d3b82ae5 app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 13:28:31 +03:00
Nikolay Khramchikhin
af994562c8 Added endpointslices discovery to k8s api (#760) 
This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21.
See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1

* Added endpointslices discovery to k8s api

Started from 1.17 k8s version endpointslices is beta,
it allows to query k8s api for endpoints more efficient.
It presents at scrape_config.yaml as separate role for kubernetes_sd_config.
kubernetes_sd_config:
- role: endpointslices

* fixed typos, changed EndpointConditions signature - with values instead of pointers
 2020-09-11 12:24:50 +03:00
Aliaksandr Valialkin
f307e6f432 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
f5cb213ef9 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin
e5c8377212 lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem 2020-09-08 19:57:20 +03:00
Nikolay Khramchikhin
fb356c434b Changed s3 configProfile flag default, (#749) 
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
 2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin
d2f6f96e4a lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699
 2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin
281d715060 lib/httpserver: add -http.connTimeout command-line flag for limiting the lifetime for incoming http connections 
This can be useful for balancing incoming connections among multiple services.
 2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin
4fa97430d7 app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
95ce89e7d7 lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling 
The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc ,
because `sw.labels` may be reset multiple times when processing big number of rows.
 2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin
bc1ca4b20b lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections 2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin
a01c56104a lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics 
This has been broken at 7785869ccc
 2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin
deff8d419a lib/promscrape: reduce memory usage when scraping targets with millions of metrics 
This should help when scraping /federate endpoints from Prometheus instances,
which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/
 2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin
5f2277624a lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy 2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin
45e770ed20 lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores 
There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.
 2020-08-28 09:49:22 +03:00
Roman Khavronenko
34ef10fbcc lib/flagutil: avoid int overflow for arch 386 (#710) 
Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32,
whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with
`int overflow` error.
 2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin
9a77ae9d1c lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels 
Previously targets with big number of metrics and/or labels could generated too big buffers,
which then could be re-used when scraping targets with small number of metrics.
This resulted in memory waste.

Now big buffers are used only for targets with big number of metrics / labels,
while small buffers are used for targets with small number of metrics / labels.
 2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin
3ea6444219 lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket 
This should reduce the number of resizings for the returned byte buffers.
 2020-08-16 22:13:38 +03:00
Roman Khavronenko
4b89da9463 lib/decimal: rename significant decimal digits to significant figures (#698) 
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding
applied to all significant figures, not just decimal ones.
 2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
1b5467f7fd lib/memory: improve log message about the memory allowed to use by VictoriaMetrics 2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin
d9f7ea1c6e lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data 
The OS should return any buffered data in the stream without the need to set the read timeout.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin
edf3ee2a9b lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
6e863376f7 lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty 
This should reduce memory allocations and copying when the byte slice is growing.
 2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin
3efa4e4e1c lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin
c6b0547847 lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin
9d79a3a99d lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches 2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin
b996280c65 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
c82a485cf6 lib/protoparser/prometheus: typo fix in error message 2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin
9e67343756 lib/promscrape: use a hint on body length instead of body capacity 
This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
 2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin
b4119bb51e lib/promscrape: reduce memory usage when scraping big number of targets 
Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
 2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin
1724cc241e lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes 2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin
60c7397be5 all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 
Such placeholders are substituted by the corresponding environment variable values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
 2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
59d95961b8 lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use 2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin
890cfe5b61 lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint 2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin
e3439a6cd0 lib/promscrape: optimize per-metric hash calculations 
This increases vmagent performance by up to 10% when scraping big number of metrics
 2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin
4ce1368e4b lib/storage: mention time range used in the query that led to error message 
This should improve detecting slow queries with too big time ranges
 2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin
f92255e803 lib/storage: mention tag filters used in the query that led to error message 
This should improve detecting invalid or heavy queries that lead to errors.
 2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin
b3d4ff7ee2 app/vmstorage: improve error logging when the request times out 2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin
e3999ac010 lib/promscrape: show real timestamp and real duration for the scape on /targets page 
Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp
 2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin
c09c881264 lib/promscrape: make errcheck happy 2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin
2dfb42a8b4 lib/promscrape: export scrape_samples_added per-target metric like Prometheus does 
This metric may be useful for detecting targets with high churn rate for the exported metrics.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683
 2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin
fd9f1463df lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails 
this is expected condition, so it is better to use WARN log level for it
 2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin
d4be3efc60 lib/promscrape: add a test for scrape config for blackbox exporter 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684
 2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin
67cacb22ac lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary 
This makes such binaries compatible with binaries from `master` branch (aka single-node version)

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
 2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin
307281e922 lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit 
This should improve data ingestion performance when heavy searches are executed

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin
dd1d59f57a lib/storage: properly check timeouts and pace limits 
Previously they were checked on every iteration for small number of iterations
 2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin
a2039b3bbc app/vmselect: return the upper bound on the number of found time series from storage.Search.Init 
This is used by a single-node version in order to reduce memory allocations during search.
See bc8381613d for details.
 2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin
b690eeff53 lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x 2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin
6c0a92a1ee lib/pacelimiter: increase scalability for multi-CPU system 2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin
13f8644f8e lib/storage: optimize prefetching metric names for the given metricIDs 2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin
f789e0fa44 lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock 2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin
a3e91c593b lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
76064ba9e7 Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer 2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin
8cc2e01386 lib/backup: allow using ~/.aws/config without region 
Use us-west-2 for determining bucket region.
 2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin
a2aa3a60eb app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
3149af624d lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 
Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this
increases query latencies since more CPUs are busy with merges.

While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.
 2020-07-31 17:53:13 +03:00
Aliaksandr Valialkin
29bbab0ec9 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
96039dcb40 lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once 
Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series,
while only a single increment is needed when inserting the first data point for this time series.
 2020-07-30 16:17:19 +03:00
Aliaksandr Valialkin
1e067401ba lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-29 13:12:35 +03:00
Sasasu
96bc476e53 lib/storage: metaindexRow use memroy more efficiently (#655) 
due to memory align the metaindexRow structure use 64-byte pre object.
this commit changes the order of field, make metaindexRow use 56-byte pre
object.

Signed-off-by: Sasasu <su@sasasu.me>
 2020-07-27 23:23:25 +03:00
Aliaksandr Valialkin
f26ef58137 lib/protoparser/prometheus: add a test for cassandra-exporter 
Thanks to Seva
 2020-07-27 18:37:46 +03:00
Aliaksandr Valialkin
94cc677b0c lib/storage: slightly reduce code difference between single-node and cluster versions 2020-07-24 01:18:05 +03:00
Aliaksandr Valialkin
fb3d1380ac lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin
dbf3038637 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin
b8303afcd8 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
7d0743422b lib/storage: properly calculate global metrics in UpdateStats() 2020-07-23 00:35:31 +03:00
Aliaksandr Valialkin
6afdcf8a20 lib/mergeset: properly calculate global metrics in UpdateStats() 
Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.
 2020-07-23 00:35:29 +03:00
Aliaksandr Valialkin
23fa44e56e lib/storage: reorder mergeBlockStreams() args in order to make them more consistent 2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin
754eac676d lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs 
This condition may occur after the following sequence of events:

1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs.
2) All the goroutines return from Storage.AddRows.
3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body.

The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal().
This may take indefinite time.
 2020-07-22 21:52:42 +03:00
Aliaksandr Valialkin
a3f48e395e app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage 2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin
67be79a0bc lib/uint64set: optimize adding items to the set via Set.AddMulti 2020-07-21 20:57:05 +03:00
Aliaksandr Valialkin
31ef39e8da lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
be0ab4fbfe lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted 
This is a follow-up commit after 12b16077c4 ,
which didn't reset the `tsidCache` in all the required places.
This could result in indefinite errors like:

    missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time

Fix this by resetting the cache inside deleteMetricIDs function.
 2020-07-14 14:05:19 +03:00
Aliaksandr Valialkin
a4c96d9e6d lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"} metric 2020-07-14 12:15:56 +03:00
Seva Poliakov
a5e713b6e0
add vm_protoparser_rows_read_total metrics to promscrape (#624) 
* add vm_protoparser_rows_read_total metrics to promscrape

move vm_protoparser_rows_read_total for promscrape to better place

move vm_protoparser_rows_read_total for promscrape to better place

* remove possibility of infinity loop at prometheus parser
 2020-07-14 12:02:25 +03:00
Roman Khavronenko
207e93b50d lib/flagutil: specify additional description for all Array type flags (#620) 
Array type flag is now defined as `value` type in flag description when printed.
This change adds additional description to every Array type flag so it would be
clear what exact type is used:
```
  -remoteWrite.urlRelabelConfig array
        Optional path to relabel config for the corresponding -remoteWrite.url
        Supports array of values separated by comma or specified via multiple flags.
```
 2020-07-13 22:00:03 +03:00
Roman Khavronenko
605711bde5 lib/persistentqueue: add vm_persistentqueue_bytes_pending metric (#619) 
Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount
of bytes in persistentqueue flushed on disk as a difference between write and read
offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes`
except of accounting for bytes in-memory.
 2020-07-13 21:54:54 +03:00
Roman Khavronenko
a02097e657 Extend metric vm_promscrape_targets with status label (#615) 
The change to `vm_promscrape_targets` metric suppose to improve observability
for `vmagent` so it will be possible to track how many targets are up or down
for every specific scrape group:
```
vm_promscrape_targets{type="static_configs", status="down"} 1
vm_promscrape_targets{type="static_configs", status="up"} 2
```
 2020-07-13 21:54:53 +03:00
Aliaksandr Valialkin
6373d377ef app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus 2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin
2012e294d1 properly calculate readCalls 2020-07-10 12:01:05 +03:00
Aliaksandr Valialkin
87f8c728bf lib/promscrape: send Accept header similar to Prometheus when scraping targets 
This should fix scraping Spring Boot servers, which return incorrect response
unless `Accept: text/plain` request header is set.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608
 2020-07-08 19:50:06 +03:00
Aliaksandr Valialkin
7335743d57 lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS 
Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense,
since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth
is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs,
i.e. to GOMAXPROCS.
 2020-07-08 17:34:27 +03:00
Roman Khavronenko
929ad74de6 lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607) 
The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to
respect the naming standard.
 2020-07-08 14:19:27 +03:00
Aliaksandr Valialkin
e401b8d527 lib/protoparser/graphite: go fmt 2020-07-08 14:13:06 +03:00
Aliaksandr Valialkin
50ecf09042 lib/protoparser/graphite: add more tests after eb45185eef 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610
 2020-07-08 14:13:03 +03:00
Seva Poliakov
1ae0334e17 Fix graphite minus one timestamp (#609) 
* fix graphite -1 timestamp

* format the graphite fix -1 timestamp
 2020-07-08 14:13:01 +03:00
Aliaksandr Valialkin
fad008df7e lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag 2020-07-08 13:54:13 +03:00
Aliaksandr Valialkin
fe58462bef lib/storage: reset MetricName->TSID cache after deleting time series 
This should prevent from adding new data points to deleted time series
without the need to check for the deleted time series.

This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set
contains big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596

Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604
 2020-07-06 22:01:24 +03:00
Aliaksandr Valialkin
77bb0e6595 lib/fs: clarify description for -fs.disableMmap command-line flag 2020-07-06 14:28:57 +03:00
Aliaksandr Valialkin
0bff96fe4b lib/storage: prioritize data ingestion over heavy queries 
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.

Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
 2020-07-05 19:44:04 +03:00
Aliaksandr Valialkin
6f1d926698 lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts 2020-07-03 21:33:48 +03:00
Aliaksandr Valialkin
ee03b4ccbd lib/promscrape: prevent from too big deadline misses on scrape retries 
The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case.
By default it is limited to scrape_interval configured for the given scrape target.
 2020-07-03 20:42:09 +03:00
Aliaksandr Valialkin
dfa83a4a35 lib/promscrape: check for nil error before checking for the returned status code when scraping targets 2020-07-03 18:37:25 +03:00
Aliaksandr Valialkin
8bb3622e9d app/vminsert: prevent from adding and/or selecting labels with empty values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
 2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin
6ebac3ab63 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
2361ad8ab4 lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-02 14:19:34 +03:00
Aliaksandr Valialkin
0f754bea49 lib/promscrape: add -promscrape.disableKeepAlive command-line flag for disabling http keep-alive connections when scraping targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-01 02:20:46 +03:00
Aliaksandr Valialkin
4cb3e7595c app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin
81e3d4305f lib/httpserver: add Unwrap method to ErrorWithStatusCode, so As and Is functions in standard errors package may properly unwrap the error inside ErrorWithStatusCode 2020-07-01 00:53:49 +03:00
Aliaksandr Valialkin
fe77d661b3 all: use errors.As instead of type assertion for detecting net.Error 2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin
0c4e8aeb2b all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode 2020-07-01 00:03:11 +03:00