Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,302 Commits 114 Branches 748 Tags 509 MiB
fa3ce450fb
Commit Graph

875 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
9e3993c585 lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. 
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
 2021-02-14 23:39:19 +02:00
Aliaksandr Valialkin
18c2075159 lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics 
These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes.

* It is better using `up == 0` query for determining failing targets.
* It is better using the following query for determining targets with exceeded limit on the number of metrics:

  scrape_samples_scraped > 0 if up == 0
 2021-02-12 05:23:27 +02:00
Aliaksandr Valialkin
4e645a5fd3 lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin
eeb92eb7fc lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata 2021-02-10 17:55:51 +02:00
Aliaksandr Valialkin
08f21d8761 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin
b27288f1b0 lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin
4262c2f7c2 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin
681dfb7485 lib/storage: fix inconsistencies in error logs 2021-02-10 16:32:21 +02:00
Aliaksandr Valialkin
148422bcba lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
17d5a03f6e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin
fa0ef143b1 lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin
5c9715a89a lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day 
This should help systems with multiple CPU cores
 2021-02-10 00:04:19 +02:00
Aliaksandr Valialkin
082eabf51e lib/fs: remove the code for tracking whether the given memory region is in page cache 
This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.
 2021-02-09 16:51:11 +02:00
Aliaksandr Valialkin
afa9cf9c57 lib/mergeset: remove dead code left after a4140de9e6 2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin
9ed7789fef optimize Storage.updatePerDateData() 2021-02-09 02:59:53 +02:00
Aliaksandr Valialkin
ea328b7391 lib/storage: skip deduplication when creating inmemory data blocks 
The deduplication will be performed later during merging such blocks.
 2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
e8ee9fa7fe app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin
2b36eb3d82 lib/cgroup: follow-up after b9bf3cbe3e 2021-02-08 16:01:26 +02:00
Nikolay
32603f57cc refactored cgroups limits, (#1061) 
adds tests, remove os.Exec
 2021-02-08 16:01:26 +02:00
Aliaksandr Valialkin
2dbb12563b lib/storage: optimize data ingestion in the beginning of every hour 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-08 12:04:51 +02:00
Aliaksandr Valialkin
1a6eb0c3cf lib/logger: exit the app if unsupported timezone value has been passed to -loggerTimezone 
While at it, clarify descrption for `-loggerTimezone` command-line flag.
 2021-02-07 23:33:35 +02:00
Aliaksandr Valialkin
c6a7288109 lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion 
This should reduce possible CPU usage spikes at the beginning of every hour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-04 18:46:23 +02:00
Aliaksandr Valialkin
b5eba70595 lib/httpserver: expose process_open_fds and process_max_fds metrics 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037
 2021-02-04 16:42:32 +02:00
Nikolay
6fc2a8e544 fixes dockerswarm (#1053) 
fixes improper usage of host network services
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028
 2021-02-04 15:57:41 +02:00
Aliaksandr Valialkin
8249f13104 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin
2976ec89b8 lib/storage: fix a bug, which breaks searching by Graphite wildcard filters 2021-02-03 20:15:50 +02:00
Aliaksandr Valialkin
45a63a1da9 sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks 2021-02-03 20:15:37 +02:00
Aliaksandr Valialkin
4b930b9ffe app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin
755b0998ce lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric 2021-02-02 16:16:51 +02:00
Aliaksandr Valialkin
4c59dbc127 lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics 2021-02-01 20:06:16 +02:00
Aliaksandr Valialkin
e79799a784 lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func 2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
fdf9de98f8 app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values 
This commit also adds --vm-round-digits command-line option to vmctl tool.
 2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
2c14c70903 lib/logger: initialize timezone by UTC in order to fix failing tests 2021-01-27 01:00:57 +02:00
Aliaksandr Valialkin
ffec5131ae lib/promscrape: export vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics 
These metrics could be useful for determining imporperly working scrape targets.
Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.
 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
4b324da947 all: consistently use timers from timerpool 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
588090765c lib/fs: properly initialize cleaner for pageCache bitmaps 
Previously it wasnt working because the timer was fired only once
 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
68a66be811 lib/logger: add -loggerTimezone command-line flag for adjusting timezone for timestamps in log messages 2021-01-26 22:53:25 +02:00
Aliaksandr Valialkin
fdced59278 lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection 2021-01-22 13:22:59 +02:00
faceair
cf76d8dd79 lib/mergeset: add missing shouldCacheBlock (#1019) 2021-01-15 11:47:25 +02:00
Aliaksandr Valialkin
81789da731 lib/backup: increase backup chunk size from 128MB to 1GB 
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
 2021-01-13 12:16:39 +02:00
Nikolay
095be83f2f bumps minimal tls version (#1012) 2021-01-13 00:39:09 +02:00
Aliaksandr Valialkin
3d79471fb3 lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here 
This is a follow-up commit after c8ea697db8
 2021-01-12 15:20:22 +02:00
Aliaksandr Valialkin
719ad49adf lib/storage: de-duplicate tags in MetricName.sortTags 
Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels
during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.
 2021-01-12 15:03:22 +02:00
Nikolay
9f0a4fd00e Fixes error handling for promscrape.streamParse (#1009) 
properly return error if client cannot read data,
properly suppress scraper errors
 2021-01-12 13:35:09 +02:00
Aliaksandr Valialkin
c97681b45c lib/promscrape: properly show scrape duration on /targets page 
Previously it has been shown as 0.000s for any scrape duration.
 2021-01-11 21:15:50 +02:00
Aliaksandr Valialkin
4ee53c3961 all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000 2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin
d5a2b120e9 app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage 2021-01-08 00:12:12 +02:00
Aliaksandr Valialkin
ca8919e8e1 lib/storage: wait for pending transactions before closing and dropping the partition 
This deflakes `make test-full-386` test
 2020-12-25 11:46:47 +02:00
Aliaksandr Valialkin
c0511144e3 lib/storage: physically remove stale parts 
Previously they were removed from partition struct, but the corresponding directories weren't removed.

This is a follow-up for 46dba00756
 2020-12-24 16:56:09 +02:00
Nikolay
dd2f815204 fixes kubernetes_sd (#983) 
* fixes kubernetes_sd,
adds missing service metadata for pod ports without endpoint
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982

* fix test
 2020-12-24 11:34:12 +02:00
Aliaksandr Valialkin
367fc17933 lib/promscrape: code prettifying for 8dd03ecf19 2020-12-24 10:57:20 +02:00
Nikolay
b00f7816e2 adds proxy_url support, (#980) 
* adds proxy_url support,
adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery,
adds proxy_url to the scrape_config for targets scrapping,
http based proxy is supported atm,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503

* fixes imports
 2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin
66f8fbbb32 lib/storage: do not remove parts outside the configured retention if they are currently merged 
These parts are automatically removed after the merge is complete.
 2020-12-24 09:02:12 +02:00
Aliaksandr Valialkin
fa3bcf220f lib/storage: remove stale parts as soon as they go outside the configured retention 
Previously such parts could remain undeleted for long durations until they are merged with other parts.
This should help for `-retentionPeriod` values smaller than one month.
 2020-12-22 19:55:07 +02:00
Aliaksandr Valialkin
6859737329 lib/storage: properly determine max rows for output part when merging small parts 2020-12-18 23:26:28 +02:00
Aliaksandr Valialkin
edbe35509e lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage 2020-12-18 20:01:20 +02:00
Aliaksandr Valialkin
1ee5a234dc lib/promscrape: remove ID field from ScrapeWork struct. Use a pointer to ScrapeWork as a key in targetStatusMap 
This simplifies the code a bit.
 2020-12-17 14:31:55 +02:00
Aliaksandr Valialkin
4fd2973e7c lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2 
Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.
 2020-12-16 23:42:17 +02:00
BigFish
60dd48c9eb lib/protoparser/prometheus/parser.go (#970) 
fix parse timestamp error if there are some whitespaces after timestamp
 2020-12-16 23:42:16 +02:00
Aliaksandr Valialkin
c4ec977934 lib/promscrape: properly remove deleted target from /targets page 
Previously `sw` variable wasn't captured correctly by the started goroutine.
 2020-12-15 20:58:37 +02:00
Aliaksandr Valialkin
eddc2bd017 lib/promscrape: properly handle scrape errors when stream parsing is enabled 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967
 2020-12-15 14:10:52 +02:00
Nikolay
7064c4eb8e adds new Array Flags (#965) 
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url

* added backward compatibility testcases for ArrayDuration and ArrayBool

* fixes bool flag

* fixes test cases
 2020-12-15 12:59:33 +02:00
Aliaksandr Valialkin
104aac170e lib/promscrape: add bootstrap styles to /targets html page 2020-12-15 12:38:29 +02:00
Aliaksandr Valialkin
ad961fe1f1 lib/promscrape: formatting fixes for /tarets page 2020-12-15 11:59:28 +02:00
Aliaksandr Valialkin
38145cfbb8 lib/promscrape: formatting fixes for /targets page 2020-12-15 11:27:22 +02:00
Aliaksandr Valialkin
d98a2f217b lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964
 2020-12-14 19:25:53 +02:00
Aliaksandr Valialkin
b904ae3722 lib/promscrape: add missing whitespace between duration and ago word at /targets page 2020-12-14 14:20:30 +02:00
Aliaksandr Valialkin
a2eb451de4 app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser 2020-12-14 14:13:01 +02:00
Nikolay
324e3aa1a5 Changes targets api (#961) 
* changes /targets api
adds html response if requester accepts text/html,
adds quick template for /targets api,
fixes pathPrefix for / requests

* changes namings

* renamed targets file

* Update app/victoria-metrics/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds trimspace to qtpl,
moves content-type for targets response closer to writer

* fixes bug with prefix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-12-14 14:13:00 +02:00
Aliaksandr Valialkin
c80d38f00c lib/promscrape/discovery/consul: reduce load on Consul API server by increasing timeout for blocking requests from 50 seconds to 9 minutes 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
 2020-12-11 17:26:34 +02:00
Aliaksandr Valialkin
a84467958a lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server 
Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg.
See https://www.consul.io/api-docs/health#list-nodes-for-service for details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170
 2020-12-08 21:53:23 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
38188e1d6b lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork() 
This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice
could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they
are no longer used.

See the related commit e205975716 and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-12-08 17:55:21 +02:00
Aliaksandr Valialkin
d5faad0240 lib/promscrape: re-use strings for labels stored in ScrapeWork 
This should reduce memory usage when working with big number of scrape targets.
 2020-12-08 12:23:44 +02:00
Aliaksandr Valialkin
06091cfdf8 lib/promscrape: export vm_promscrape_scrapers_{started|stopped}_total metrics for monitoring target churn rate 2020-12-08 11:58:44 +02:00
Aliaksandr Valialkin
affcee199c lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value 
This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.
 2020-12-08 11:52:20 +02:00
Aliaksandr Valialkin
56a0b058c1 lib/promscrape: export vm_promscrape_active_scrapers{type="<sd_type>"} metric for tracking the number of active scrapers per each service discovery type 2020-12-08 01:54:44 +02:00
Aliaksandr Valialkin
b5b32c65b0 lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun command-line flag is passed 
Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944
 2020-12-07 13:18:16 +02:00
Aliaksandr Valialkin
9d79d11a6c lib/promscrape: mention in scrape error message that scrape errors can be disabled by -promscrape.suppressScrapeErrors command-line flag 2020-12-06 23:27:07 +02:00
Aliaksandr Valialkin
f6d32f99d7 lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set 2020-12-06 13:19:32 +02:00
Aliaksandr Valialkin
3d00613076 lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol 2020-12-06 12:00:28 +02:00
Aliaksandr Valialkin
1430bbcf33 lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries 
Too low limit could result in unexpected errors when performing big number of blocking queries.
 2020-12-05 12:15:47 +02:00
Aliaksandr Valialkin
528587deef lib/flagutil: make golangci-lint happy by using strings.TrimPrefix instead of manual prefix removal via strings.HasPrefix 2020-12-03 22:07:26 +02:00
Aliaksandr Valialkin
bdac2171f1 all: do not print usage info for all the flags when incorrect command-line flag is passed 
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
 2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin
96190f9d45 lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher 2020-12-03 20:14:48 +02:00
Aliaksandr Valialkin
4e4a93c586 lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header 2020-12-03 20:05:54 +02:00
Aliaksandr Valialkin
7a889f6850 lib/promscrape: code cleanup after c6dee6c52d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
 2020-12-03 19:52:09 +02:00
Nikolay
0b302d33cb Changes consul discovery api (#921) 
* adds consul watch api,
it must reduce load on consul service with blocking wait requests,
changed discoveryClient api with fetchResponseMeta callback.

* small fix

* fix after master merge

* adds watch client at discovery utils

* fixes consul watcher,
changes namings,
fixes data race

* small typo fix

* sanity fix

* fix naming and service node update
 2020-12-03 19:52:08 +02:00
Aliaksandr Valialkin
9eca96596f lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test 2020-11-29 01:25:50 +02:00
Aliaksandr Valialkin
2385ac11c0 lib/promscrape: fix failing tests after a906b3862f 2020-11-29 01:25:49 +02:00
Aliaksandr Valialkin
2ed721e457 lib/protoparser/prometheus: properly parse OpenMetrics timestamps 
OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds.
This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.
 2020-11-27 14:54:36 +02:00
Aliaksandr Valialkin
3bb9bf33d6 lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets 2020-11-26 18:32:16 +02:00
Aliaksandr Valialkin
af667c59c1 all: typo fix: thouthand->thousand 2020-11-26 13:34:05 +02:00
Aliaksandr Valialkin
3dd2282ed9 lib/promscrape: release http response non-200 status code 2020-11-26 13:25:25 +02:00
Aliaksandr Valialkin
3f52e59efe app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun 2020-11-25 23:01:39 +02:00
Aliaksandr Valialkin
8f8727cb65 lib/mergeset: tune the number of rawItemsBlocks to merge at once 
512 blocks give higher ingestion performance and slightly lower memory usage
 2020-11-25 21:53:15 +02:00
Aliaksandr Valialkin
8fcd87a6a5 lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset 2020-11-25 21:20:02 +02:00
Aliaksandr Valialkin
03002f1fe1 lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention 
This should simplify debugging when the source of the metric with unexpected timestamp must be found.
 2020-11-25 14:44:29 +02:00
Aliaksandr Valialkin
4848a05924 lib/storage: typo fix in error message: allowd->allowed 2020-11-25 14:15:54 +02:00