2446 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	5afa54e845	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() ... This improves the API consistency	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	78f9a8aafd	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index ... Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit 55e7afae3a, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:56:55 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability ... 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin	7fc03a1deb	app/vmagent/remotewrite: add -remoteWrite.header command-line flag for setting additional http headers to send to -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin	4fb0f15322	all: readability improvements for query traces ... - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:19:43 +03:00
ttyv	00956e585d	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786) ... Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	7d5d33fd71	lib/storage: return marshaled metric names from SearchMetricNames ... Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin	15da802f5f	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:52:39 +03:00
Aliaksandr Valialkin	399d4c36ae	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin	64505e924d	app/vmstorage: extract vmselect api server into a separate package - lib/vmselectapi ... This opens doors for implementing vmselect api server at vmselect level, so top-level vmselect could query lower-level vmselect nodes in the same way as it queries vmstorage nodes. This will create the ability to create highly available querying architecture when multiple independent VictoriaMetrics clusters with the same data are located in distinct availability zones. In this case we can use top-level vmselect instead of Promxy for simultaneous querying of all the clusters in all the AZs.	2022-06-27 14:20:41 +03:00
Aliaksandr Valialkin	6386f117c8	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin	926fccbb8d	lib/storage: add querytracer to more contexts ... querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 12:53:49 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them ... This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	71b0dfdefa	lib/promscrape: always send stale markers with the real scrape timestamp ... This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets ... This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	fe2269b999	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package ... This package already has the same name, so there is no need in explicit name	2022-06-21 20:24:28 +03:00
Loki's Wager	ca4730c00f	BugFix part_header.go (#2763) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin	288d13af8d	lib/netutil: parallelize background pings for remote addresses ... This should improve the time needed for determining unavailale remote addresses across big numer of ConnPool's. This is a follow-up for a1629bd3be Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-21 13:32:27 +03:00
Aliaksandr Valialkin	a1629bd3be	lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful ... If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped until the background goroutine determines that the given address can be successfully dialed. This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711 This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756 The main differences are: - The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool. This makes possible re-using this feature everywhere lib/netutil.ConnPool is used. - The check doesn't take into account handshake errors for already established connections. Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.	2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin	45e9732764	docs: follow-up after e4d6b750f6	2022-06-20 17:15:52 +03:00
Nikolay	15662c0f29	lib/httpserver: adds flagsAuthKey command-line flag (#2758) ... * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:15:51 +03:00
Aliaksandr Valialkin	b28c6febf9	app/{vminsert,vmselect}: add -vmstorageDialTimeout command-line flag for tuning the maximum time needed for establishing connections to vmstorage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-20 15:17:34 +03:00
Aliaksandr Valialkin	270ad39359	lib/storage: properly take into account already registered series when -storage.maxHourlySeries or -storage.maxDailySeries limits are enabled ... The commit 5fb45173ae takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:53:41 +03:00
Aliaksandr Valialkin	7a79e7c0ef	lib/storage: create per-day indexes together with global indexes when registering new time series ... Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:32:41 +03:00
Aliaksandr Valialkin	88e1221b35	lib/storage: do not register new series if -storage.maxHourlySeries or -storage.maxDailySeries limits are exceeded ... Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:03:02 +03:00
Aliaksandr Valialkin	c5ac176153	lib/storage: reset metric id caches for the previous and the current hour ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:02:51 +03:00
Aliaksandr Valialkin	450aa0ae5a	lib/promrelabel: support action: graphite relabeling ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:25:49 +03:00
Aliaksandr Valialkin	45fa9d798d	app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb	2022-06-14 18:39:00 +03:00
Aliaksandr Valialkin	fb77843639	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:34:13 +03:00
Aliaksandr Valialkin	3167fbc21d	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:28:21 +03:00
Nikolay	e23af8f05c	lib/httpserver: backport changes from master branch (#2697) ... * lib/httpserver: backport changes from master branch adds basicAuth adds authKey check for /metrics and /debug/pprof requests it should improve security for cluster components * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-14 13:02:44 +03:00
Aliaksandr Valialkin	4af43a4a75	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:28:37 +03:00
Aliaksandr Valialkin	61e03f172b	app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them	2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin	cb39eada77	all: improve query tracing coverage for indexdb search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:04:02 +03:00
Howie	4afd7aa695	feat: rule limit (#2676) ... vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 13:15:33 +03:00
Aliaksandr Valialkin	a9ea3fee38	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:16:12 +03:00
Dmytro Kozlov	f2754c3e90	Cardinality explorer (#2625) ... * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:54:27 +03:00
Roman Khavronenko	2b5e1dee91	vmagent: update SD duration histogram metric if SD is active (#2677) ... The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:53:06 +03:00
Roman Khavronenko	5f33445f66	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:02:55 +03:00
Aliaksandr Valialkin	b6e3c12811	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string ... This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin	68b6ddfb14	all: follow-up after 8edb390e21 ... - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin	3dbb19d624	lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534 ... - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin	dd0d773c13	lib/promauth: follow-up after 006b8c7534 ... - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	6c2fb9d8c4	lib/promscrape: add -promscrape.cluster.name command-line flag ... This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 01:11:23 +03:00
Dmytro Kozlov	ce8aade80e	lib/promscrape: adds service discovery visualization for /targets page(#2675) ... * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-04 01:11:23 +03:00
Nikolay	72e43ef2fe	lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 (#2672)	2022-06-04 01:11:23 +03:00
hadesy	28d4624f60	promscrape/discovery: support kubeconfig (#2533)	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	cc226e6ebe	docs/CHANGELOG.md: follow-up after 2177089f94	2022-06-01 14:57:39 +03:00
Roman Khavronenko	e9ee043879	lib/storage: make indexdb/tagFilters cache size configurable (#2667) ... The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Roman Khavronenko	bca90d7148	promrelabel: add support of lowercase and uppercase relabeling actions (#2665) ... * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Aliaksandr Valialkin	fedfc9e686	lib/storage: stop background merge when storage enters read-only mode ... This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin	afced37c0b	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin	386f6110ec	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum ... In this case there is no need in converting int64 to int	2022-06-01 01:43:25 +03:00
Aliaksandr Valialkin	945e9fa8c4	lib/storage: make fmt	2022-05-31 12:42:48 +03:00
Aliaksandr Valialkin	727cc119b6	lib/storage: do not take into account series from the next day when match[] filter is passed to /api/v1/status/tsdb	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	cd1fa2e4cd	issue-2594: use embedded for static files (#2650) ... embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	6add79143b	removed redundant return (fixed linter) (#2647) ... * removed redundant return * updated lint package version	2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin	f149d56ac2	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag ... This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 23:00:30 +03:00
Aliaksandr Valialkin	38beb9fe04	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag ... This is a follow-up for 0fbf59199a See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:07:14 +03:00
阳明	e4df648ea0	lib/storage: Remove the effect of time zone on next retention period (#2568) (#2574)	2022-05-25 15:10:19 +03:00
Roman Khavronenko	7406665fc3	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) ... * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:17:21 +03:00
Boris Petersen	3a8b4fab97	Add ability to sign requests for all AWS services (#2604) ... This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin	116c0b8f2e	docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag	2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin	d8a276fbe4	lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() ... This should reduce potential spikes in the number of established connections in the following cases: - when the connection establishing procedure becomes temporarily slow - after a temporary spike in the rate of ConnPool.Get() calls See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552	2022-05-11 14:11:06 +03:00
Aliaksandr Valialkin	0d0561ca8c	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future ... Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() ... Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit 67b10896d2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin	af0da45d3e	lib/promscrape: rename promscrape_stale_samples_created_total metric to vm_promscrape_stale_samples_created_total, so its name is consistent with the rest of vm_promscrape_ metrics	2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin	9d40bb7137	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section	2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin	2ce1d09135	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call ... Previously filters wheren't passed to this call after the commit 0e09fdb8b0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin	873f55bac5	lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query ... The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: 0e09fdb8b0 d289ecded1	2022-05-05 10:29:47 +03:00
Dmytro Kozlov	4f40dc9829	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535) ... * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:48 +03:00
Nikolay	7e58cba6cf	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458) ... * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:28:37 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530) ... standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Aliaksandr Valialkin	361b08c30e	lib/storage: leave the last sample per each discrete interval during the deduplicaton ... This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin	190c8b463c	lib/netutil: close connections in ConnPool if they are idle for more than 30 seconds ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-05-02 15:01:52 +03:00
Artem Navoiev	11db05a4ff	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487) ... * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:24:12 +03:00
Aliaksandr Valialkin	a436836402	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:58:08 +03:00
Dima Lazerka	837e440865	Fix targetstatus qtpl paths (#2517) ... Ran `make quicktemplate-gen` from the root directory	2022-04-29 11:18:14 +03:00
Aliaksandr Valialkin	aa82987d70	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing ... This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin	a85ef60b4b	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:24:13 +03:00
Aliaksandr Valialkin	4c3cd96db5	lib/promauth: add support for min_version option at tls_config section in the same way as Prometheus does	2022-04-23 00:24:11 +03:00
Aliaksandr Valialkin	808a2f3b61	lib/promauth: add support for proxy_url option at oauth2 section in the same way as Prometheus does	2022-04-23 00:01:53 +03:00
Aliaksandr Valialkin	4ade8511e2	lib/promauth: add support for tls_config section at oauth2 config in the same way as Prometheus does	2022-04-23 00:01:52 +03:00
Aliaksandr Valialkin	c2b13e6a04	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects ... Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin	a89e31b304	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does ... See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin	cc6eae6992	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems ... Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin	60f74dab56	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated ... This is a follow-up after 26b78ad707	2022-04-22 13:23:37 +03:00
Aliaksandr Valialkin	ed1b394a1a	app/vmstorage: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page ... These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin	fea9d1e6ee	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for 2341bd48d7	2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin	1e0517b9cd	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:41:23 +03:00
Aliaksandr Valialkin	1ae16bf671	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:27 +03:00
Aliaksandr Valialkin	e9f08b1e6a	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects ... There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin	909a3ee0e4	lib/promscrape: follow-up after 91e290a8ff	2022-04-20 16:12:26 +03:00
Nikolay	429848a67d	lib/promscrape: reduce latency for k8s GetLabels (#2454) ... replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:12:25 +03:00
Dmytro Kozlov	9dbfd99777	lib/promscrape: simply update UI (#2479) ... * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 15:38:04 +03:00
Aliaksandr Valialkin	45385a5dc6	lib/promscrape: optimize getScrapeWork() function ... Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:34:18 +03:00
Aliaksandr Valialkin	bfa0b8f710	lib/promscrape: use a hash over target labels as a key for dropped targets' map ... This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:23:54 +03:00
Aliaksandr Valialkin	d0bac8e224	all: typo fix: Kuberntes -> Kubernetes	2022-04-20 10:51:41 +03:00
Dmytro Kozlov	17552dba8b	lib/promscrape: Enable filters for endpoint and labels (#2466) ... * lib/promscrape: Enable filters for endpoint and labels * lib/promscrape: cleanup * lib/promscrape: update template * lib/promscrape: move logic filter logic to backend * lib/promscrape: updated placeholder * lib/promscrape: updated placeholder * lib/promscrape: use two different fields for filters, updated form, added error on parsing queries * lib/promscrape: rename functions * lib/promscrape: removed unused values * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-19 18:27:44 +03:00
Nikolay	628905f080	lib/promscrape: adds job restart method (#2455) ... * lib/promscrape: adds job restart method it must restart only ScrapeConfig with changed content this change greatly reduce time, that needed for job restart and it should decrease possible data loss when config frequently changed at kubernetes based deployments Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 20:29:33 +03:00
Aliaksandr Valialkin	7debf57ca6	lib/httpserver: clarify that -tls flag enables TLS for http requests to -httpListenAddr	2022-04-16 16:59:41 +03:00
Aliaksandr Valialkin	a7689e1b0c	app/vmstorage: add support for mTLS cipher suites via -cluster.tlsCipherSuites command-line flag ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404	2022-04-16 16:36:38 +03:00
Aliaksandr Valialkin	27e74f25d6	lib/httpserver: follow up after def0032c7d	2022-04-16 15:52:44 +03:00
Dmytro Kozlov	26ae50ec26	lib/httpserver: added tlsCipherSuites flag (#2468) ... * lib/httpserver: added tlsCipherSuites flag * lib/httpserver: compare lower case strings * lib/httpserver: use EqualFold * lib/httpserver: used flagutil.NewArray, supported only strings cipher suites * lib/httpserver: updated flag description, added flag to documentation * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 15:52:42 +03:00
Aliaksandr Valialkin	c50e48a74c	lib/promscrape: follow-up after baa1c24b36	2022-04-16 14:26:38 +03:00
Nikolay	a56ee034af	lib/promscrape: removes omitempty for ScrapeConfig (#2457) ... This change fixes incorrect marshalling for ScrapeConfig it affects http endpoint and ScrapeConfig checksum. With omitempty, custom Marshaller is not called if field is not a pointer. Previously this issue happened at vmalert	2022-04-16 14:26:36 +03:00
Aliaksandr Valialkin	4a3172f150	lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* ... This allows Go complier to generate more optimal code without bound checks	2022-04-12 12:56:52 +03:00
Aliaksandr Valialkin	70ad171070	lib/promscrape: follow-up after 7e79adfb55	2022-04-12 12:37:03 +03:00
Nikolay	e26bcb8bbb	lib/promscrape: allows to use k8s pod name as clusterMemberNum (#2436) ... * lib/promscrape: allows to use k8s pod name as clusterMemberNum it must improve user expirience and simplify clustering scrapers. it must allow to use vmagent cluster with distroless images https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-12 12:37:02 +03:00
Aliaksandr Valialkin	81b7a31cb1	app/vmstorage: properly handle maxSeries limit passed from vmselect to vmstorage	2022-04-12 11:19:07 +03:00
Aliaksandr Valialkin	e3bf464f11	lib/protoparser/native: follow-up after fe01f4803d	2022-04-11 19:27:53 +03:00
Nikolay	39225fc809	lib/protoparser/native: fixes parseStream dead-lock (#2423) ... previously, if native block cannot be unmarshaled, wg.Done wasn't called by unmarshal work. It leads to connection blocking and possible dead-lock at client side	2022-04-11 19:27:51 +03:00
Aliaksandr Valialkin	edb139cfe4	lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to ... This metric is equivalent to `vm_available_memory_bytes`, but it has better name, since the metric is related to a process, not VictoriaMetrics itself. Leave `vm_available_memory_bytes` for backwards compatibility.	2022-04-07 15:24:08 +03:00
Aliaksandr Valialkin	cb319b15bb	lib/storage: increase the number of rawRowsShard shards on systems with more than 4 CPU cores ... This should improve data ingestion scalability on systems with many CPU cores	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	8ef9348801	lib/mergeset: use more rawItemsShard shards on multi-CPU systems ... This should improve the scalability for registering of new time series on multi-CPU system	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	db00ddd23e	lib/mergeset: skip common prefixes when comparing inmemoryBlock items ... This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData if they have common prefix. While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items. This should improve performance for inmemoryBlock.MarshalSortedData during background merge.	2022-04-06 18:55:25 +03:00
Aliaksandr Valialkin	88c2631320	lib/protoparser: remove superflowous memory allocations during protocol parsing	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	123a88bb65	lib/storage: reuse sync.WaitGroup objects ... This reduces GC load by up to 10% according to memory profiling	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	f526c7814e	lib/cgroup: reduce the default GOGC value from 50% to 30% ... This reduces memory usage under production workloads by up to 10%, while CPU spent on GC remains roughly the same. The CPU spent on GC can be monitored with go_memstats_gc_cpu_fraction metric	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	0f1ebd911d	lib/workingsetcache: reuse prev cache after its reset ... This should reduce memory churn rate	2022-04-05 20:39:44 +03:00
Aliaksandr Valialkin	ac93c36be7	lib/workingsetcache: check more frequently for cache size overflow ... This should reduce the probability of cache size limit overflow	2022-04-05 18:05:33 +03:00
Nikolay	7eb49d204f	vmctl verify-blocks command (#2390) ... * lib/protoparser: changes ParseStream for native format uses reader instead of http.Request updates app/vmagent and app/vmagent method usage * app/vmctl: add verify-block subcommand it allows to check exported from VictoriaMetrics data block in native format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362 Update app/vmctl/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-04-05 17:46:36 +03:00
Aliaksandr Valialkin	fca0cb8156	lib/workingsetcache: reduce the expiration duration from 20 minutes to 10 minutes ... This should reduce memory usage for the cache under high churn rate	2022-04-05 17:08:43 +03:00
Aliaksandr Valialkin	8752cce157	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage ... This reduces the max memory usage for vminsert and vmstorage under heavy ingestion rate by up to 50% on production workload	2022-04-05 15:39:58 +03:00
Nikolay	4cf6219e07	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293) ... * lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache It should decrease memory usage for regexp caching with storing cacheEntry by pointer - golang map should be able to effectivly shrink it's size original issue with this case - unexpected map grows and storage OOM Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Adds missing metrics for regexp cache and regexpPrefixes cache * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 12:57:27 +02:00
Aliaksandr Valialkin	b843f0e229	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:28:14 +02:00
Aliaksandr Valialkin	a8a4581c37	lib/blockcache: properly remove references to deleted parts ... Previously references to deleted parts may remain active as cache.m keys. This could prevent from proper memory de-allocation. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This is a follow-up for 88605a7ea2	2022-03-18 17:07:54 +02:00
Aliaksandr Valialkin	e35c9124b7	lib/storage: reduce the interval for checking for free disk space from 30 seconds to 1 second ... This should reduce the probability of out of disk space panics when -storage.minFreeDiskSpaceBytes is set to low values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2305	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	7c92aaeaa4	lib/blockcache: properly release memory occupied by deleted entries ... Proviously the deleted entries could remain referenced via lastAccessHeap for long time. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	a6d65fc824	lib/storage: typo fix after e7831ae154	2022-03-18 16:53:19 +02:00
jduncan0000	e7831ae154	Fix for issue #2255 - matchTagFilters for positive empty-match filters (#2304) ... * fix for issue 2255 - matchTagFilters for positive empty-match filters * add example to comments * formatting * add test for positive empty match * formatting	2022-03-18 13:08:54 +02:00
Aliaksandr Valialkin	698458b742	lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig	2022-03-17 19:46:20 +02:00
Aliaksandr Valialkin	191977b324	lib/storage: trashing -> thrashing typo in docs ... This is a follow-up for 918ed5cb32	2022-03-16 13:28:29 +02:00
Vic (Shihang) Li	9767fcd837	fix: change thrashing typo (#2317)	2022-03-16 13:05:55 +02:00
Aliaksandr Valialkin	3f999b11f7	lib/mergeset: remove aux buffers from inmemoryPart ... This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf68da79e	lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock ... This should improve performance when registering new time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf4f7bf21	lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	f4e466955d	lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool ... The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart, so sync.Pool usage should reduce overall memory usage and improve performance because of better locality of reference when marshaling inmemoryBlock to inmemoryPart. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	b47f18f555	lib/{mergeset,storage}: tune compression levels for small blocks ... This should reduce CPU usage spent on compression	2022-02-25 15:34:13 +02:00
Aliaksandr Valialkin	28b610db07	lib/storage: document why job-like and instance-like labels must be stored at mn.Tags[0] and mn.Tags[1] ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2244	2022-02-25 13:21:53 +02:00
Aliaksandr Valialkin	d1881fa582	lib/storage: add a comment to indexSearch.containsTimeRange() on why it allows false positives ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2239	2022-02-24 12:48:33 +02:00
Aliaksandr Valialkin	02a922b53f	lib/storage: properly handle series selector matching multiple metric names plus a negative filter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238 This is a follow-up for 00cbb099b6	2022-02-24 12:11:53 +02:00
Aliaksandr Valialkin	1967b9c211	lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort() ... There is no need to sort the underlying data according to sorted items there. This should reduce cpu usage when registering new time series in `indexdb`. Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245	2022-02-24 11:19:44 +02:00
Aliaksandr Valialkin	2431c9cf81	lib/promrelabel: add support for conditional relabeling via if filter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998	2022-02-24 02:35:13 +02:00
Aliaksandr Valialkin	6fd85117ac	lib/workingsetcache: do not rotate cache if it is in whole state ... This should reduce the maximum memory usage for the cache in `whole` state	2022-02-23 22:55:10 +02:00
Aliaksandr Valialkin	244c23ea2c	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:42:27 +02:00
Aliaksandr Valialkin	fcaa0c5202	lib/storage: optimize /api/v1/status/tsdb call by skipping all the artificially created tag entries at once ... This is a follow-up for b71be42d90	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	986c12487a	lib/mergeset: typo fix after b6ed9afd6d	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	d4df32cc6b	lib/blockcache: evict entries from the cache in LRU order ... This should improve hit rate for smaller caches	2022-02-21 19:00:04 +02:00
Roman Khavronenko	5a4b16794d	Consul SD - update services on the watcher's start (#2202) ... * lib/discovery/consul: update services on the watcher's start Previously, watcher's start was only initing goroutines for discovery but not waiting for the first iteration to end. It means first Consul discovery wasn't returning discovered targets until the next iteration. The change makes the watcher's start blocking until we get first discovery iteration done and all registries updated. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: remove workarounds for consul SD Now when consul SD lib properly updates services on the first start, we don't need workarounds in vmalert. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/discovery/consul: update after review Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:33:33 +02:00
Roman Khavronenko	bd7837d524	lib: allow to configure cache size by type (#2206) ... * lib: allow to configure cache size by type https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 13:55:51 +02:00
Aliaksandr Valialkin	5260d7a954	lib/storage: typo fix after c3affb0c4f	2022-02-17 12:56:33 +02:00
Aliaksandr Valialkin	d9bdb42219	lib/storage: simplify code for searching for label values ... This is a follow-up after 9dd191b27c	2022-02-17 12:39:14 +02:00
Aliaksandr Valialkin	2ebc3d21c3	lib/storage: properly skip composite tag entries when searching for tag names or tag values ... This is a follow-up for b71be42d90 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 23:02:18 +02:00
Aliaksandr Valialkin	3107224306	lib/blockcache: fix TestCache by ensuring that the cache size can be divided by the number of cache shards ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2204	2022-02-16 18:48:09 +02:00
Aliaksandr Valialkin	63bc89dd81	lib/storage: document why tsid cache is reset before saving it to disk ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2205	2022-02-16 18:37:29 +02:00
Aliaksandr Valialkin	ee066aa0d5	lib/storage: use binary search instead of full scan for skipping artificial tags when searching for tag names or tag values ... This should improve performance for /api/v1/labels and /api/v1/label/<label_name>/values See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 18:17:27 +02:00
Roman Khavronenko	8cbcc560b9	vmagent: fix js error on CollapseAll/ExpandAll buttons click (#2192) ... * vmagent: fix js error on CollapseAll/ExpandAll buttons click `Uncaught TypeError: Cannot read properties of null (reading 'style')` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-15 12:54:39 +02:00
Corporte Gadfly	cf8df24227	match fileSDCheckInterval with prometheus file_sd_config default (#2188)	2022-02-15 12:05:57 +02:00
Aliaksandr Valialkin	5d8ea8c918	docs/CHANGELOG.md: document 3d890e89f1	2022-02-14 17:42:33 +02:00
Nikolay	748034e7af	Adds server certificate reload for lib/http (#2186) ... * Adds server certificate reload for lib/http https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171 * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-14 17:42:33 +02:00
Nikolay	c11d0949c8	fixes all_tenants query option usage for openstack service discovery (#2184) ... explicit use configuration parametr instead of conditional add https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182	2022-02-14 13:13:03 +02:00
Aliaksandr Valialkin	31b42e9c57	lib/promscrape: add expand all and collapse all buttons to /targets page ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-12 18:42:01 +02:00
Aliaksandr Valialkin	53c2135d2a	lib/storage: tune the logic for pre-populating of the per-day inverted index for the next day ... - Postpone the pre-poulation to the last hour of the current day. This should reduce the number of useless entries in the next per-day index, which shouldn't be created there, when the corresponding time series are stopped to be pushed during the current day. - Make the pre-population more smooth in time by using the hash of MetricID instead of MetricID itself when calculating the need for for the given MetricID pre-population. - Sync the logic for pre-population of the next day inverted index with the logic of pre-populating tsid cache after indexdb rotation. This should improve code maintainability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401	2022-02-12 16:39:33 +02:00
artifactori	943fc056ca	Show gce sdconfig zone on vmagent:8429/config (#2178) ... * vmagent: add test for marshalling gce sdconfig with ZoneYAML * vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig	2022-02-12 00:48:38 +02:00
Roman Khavronenko	d107f86fbc	lib/index: reduce read/write load after indexDB rotation (#2177) ... * lib/index: reduce read/write load after indexDB rotation IndexDB in VM is responsible for storing TSID - ID's used for identifying time series. The index is stored on disk and used by both ingestion and read path. IndexDB is stored separately to data parts and is global for all stored data. It can't be deleted partially as VM deletes data parts. Instead, indexDB is rotated once in `retention` interval. The rotation procedure means that `current` indexDB becomes `previous`, and new freshly created indexDB struct becomes `current`. So in any time, VM holds indexDB for current and previous retention periods. When time series is ingested or queried, VM checks if its TSID is present in `current` indexDB. If it is missing, it checks the `previous` indexDB. If TSID was found, it gets copied to the `current` indexDB. In this way `current` indexDB stores only series which were active during the retention period. To improve indexDB lookups, VM uses a cache layer called `tsidCache`. Both write and read path consult `tsidCache` and on miss the relad lookup happens. When rotation happens, VM resets the `tsidCache`. This is needed for ingestion path to trigger `current` indexDB re-population. Since index re-population requires additional resources, every index rotation event may cause some extra load on CPU and disk. While it may be unnoticeable for most of the cases, for systems with very high number of unique series each rotation may lead to performance degradation for some period of time. This PR makes an attempt to smooth out resource usage after the rotation. The changes are following: 1. `tsidCache` is no longer reset after the rotation; 2. Instead, each entry in `tsidCache` gains a notion of indexDB to which they belong; 3. On ingestion path after the rotation we check if requested TSID was found in `tsidCache`. Then we have 3 branches: 3.1 Fast path. It was found, and belongs to the `current` indexDB. Return TSID. 3.2 Slow path. It wasn't found, so we generate it from scratch, add to `current` indexDB, add it to `tsidCache`. 3.3 Smooth path. It was found but does not belong to the `current` indexDB. In this case, we add it to the `current` indexDB with some probability. The probability is based on time passed since the last rotation with some threshold. The more time has passed since rotation the higher is chance to re-populate `current` indexDB. The default re-population interval in this PR is set to `1h`, during which entries from `previous` index supposed to slowly re-populate `current` index. The new metric `vm_timeseries_repopulated_total` was added to identify how many TSIDs were moved from `previous` indexDB to the `current` indexDB. This metric supposed to grow only during the first `1h` after the last rotation. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-12 00:34:44 +02:00
Aliaksandr Valialkin	21a42e990f	lib/storage: fix broken BenchmarkHeadPostingForMatchers for {i=~".*"} after f4dead529f ... The commit f4dead529f makes such query to return nothing instead of all the time series. This aligns more with Prometheus behaviour.	2022-02-12 00:28:21 +02:00
Roman Khavronenko	791cad8c2e	lib/promscrape: support prometheus-like duration in scrape configs (#2169) ... * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:51 +02:00
Aliaksandr Valialkin	727b29d4a3	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpointslice_{label,annotation}* labels to be consistent with other role values for Kubernetes service discovery	2022-02-11 14:56:10 +02:00
Nikolay	265938a385	fixes service discovery for kubernetes (#2173) ... * fixes service discovery for kubernetes now it must take in account all pods that belong to the discovered endpoint and endpointslice adds simple test for endpoints https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134 * wip * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin	895c9f4f11	lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin	102c9a4bf9	lib/blockcache: use higher number of shards for higher number of CPU cores ... This should reduce mutex contention and increase performance Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:11 +02:00
Aliaksandr Valialkin	ad19c9d302	lib/promscrape: fix errors in test config ... The errors were discovered after enabling strict parse mode by default. See 9bb60ab00f	2022-02-08 20:10:28 +02:00
Aliaksandr Valialkin	fae3040868	lib/blockcache: split the cache into multiple shards ... This should reduce contention on cache mutex on hosts with many CPU cores, which, in turn, should increase overall throughput for the cache. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-08 19:48:32 +02:00
Aliaksandr Valialkin	a0a56d6c1c	lib/mergeset: tune sizes for indexdb/dataBlocks and indexdb/indexBlocks according to production workload ... This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742	2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin	eed66b6640	lib/promscrape: set -promscrape.config.strictParse to true by default ... This allows detecting long-living silent errors in -promscrape.config	2022-02-08 15:42:33 +02:00
Aliaksandr Valialkin	8863339b6b	lib/blockcache: make fmt	2022-02-08 15:42:31 +02:00
Aliaksandr Valialkin	1caee74235	lib/blockcache: eliminate possible race when Cache.Put is called for the same entry from multiple goroutines ... The race could result in incorrect cache size tracking, which, in turn, could result in too frequent cache cleaning	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	10476738a8	lib/blockcache: increase the lifetime for rarely accessed blocks from 2 minutes to 5 minutes ... This should improve data ingestion speed if time series samples are ingested with interval bigger than 2 minutes. The actual interval could exceed 2 minutes if the original interval between samples doesn't exceed 2 minutes in the case of slow inserts. Slow inserts may appear in the following cases: * Big number of new time series are pushed to VictoriaMetrics, so they couldn't be registered in 2 minutes. * MetricName->tsid cache reset on indexdb rotation or due to unclean shutdown. In this case VictoriaMetrics needs to load MetricName->tsid entries for all the incoming series from IndexDB. IndexDB uses the block cache for increasing lookup performance. If the cache has no the needed block, then IndexDB reads and unpacks the block from disk. This requires an extra disk read IO and CPU. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This also should increase performance for periodically executed queries with intervals from 2 minutes to 5 minutes. See the previous similar commit - 43103be011 It is possible that the timeout can be increased further. Let's collect production numbers for this change so the timeout could be adjusted further.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	b7cefff7b0	lib/workingsetcache: use the original cache size limits when rotating caches ... Previously limits for new caches were taken from cache stats. These limits could mismatch the original limits. This could result in failed cache load if the stored cache has been created with the limits obtained from cache stats.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	87071640a7	lib/blockcache: return proper number of entries from the cache ... This has been broken in 0d7374ad2f	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	34d14c4940	all: substitute zeroTime with time.Time{}, since this generates more optimal binary code	2022-02-07 14:36:41 +02:00
Aliaksandr Valialkin	e2d12a25e0	lib/netutil: increase dial timeout from 1 second to 5 seconds ... There are real-world cases when TCP connection needs more than 1 second to be established.	2022-02-07 12:33:40 +02:00
Aliaksandr Valialkin	d24e5d9efd	lib/promscrape: show the total number of scrapes and the total number of scrape errors per target at /targets page ... This information may be useful when debugging unreliable scrape targets	2022-02-03 20:23:27 +02:00
Aliaksandr Valialkin	678b3e71db	lib/promscrape: provide the ability to fetch target responses on behalf of vmagent or single-node VictoriaMetrics ... This feature may be useful when debugging metrics for the given target located in isolated environment	2022-02-03 19:02:12 +02:00
Aliaksandr Valialkin	5f266370c5	all: follow-up after 4bdd10ab90 ... Properly use new bytesutil.Resize* functions	2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin	d8d59ff760	lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	02b2bfcff3	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations ... Follow-up for f4989edd96 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	084664d780	lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64()	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	0fbfa8c245	lib/storage: avoid allocations of tsidPrev on every blockStreamReader.NextBlock() call ... This is a follow-up for 00b7c97d2a Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-31 22:47:16 +02:00
Aliaksandr Valialkin	a02dde6cc7	lib/cgroup: fall back to runtime.NumCPU() when determining process_cpu_cores_available metric if it is impossible to determine cpu quota via cgroups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:31:12 +02:00
Aliaksandr Valialkin	566e12874d	lib/cgroup: expose process_cpu_cores_available metric ... This metric shows the number of CPU cores available to the process. This allows creating alerting rules on CPU saturation with the following query: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	776b7bc9f8	lib/storage/table.go: add missing tb.ptwsLock.Unlock() before the return ... This is a follow-up for a1083d0531 See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2103	2022-01-28 12:12:58 +02:00
匠心零度	a1083d0531	optimized code (#2103) ... * optimized code ,because only the first error,so no need var errors []error * optimized code ,because only the first error,so no need var errors []error Co-authored-by: lirenzuo <lirenzuo@shein.com>	2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin	6232eaa938	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions ... Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin	7ec0705b98	lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock ... This should reduce the memory required for indexdb/dataBlocks cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin	65176726b3	lib/logger: removed broken test after 746ee191e8	2022-01-24 12:15:11 +02:00
Aliaksandr Valialkin	49650fe6aa	lib/logger/throttler.go: show the original location of the error and warning message ... Previously the location inside LogThrottler implementation was shown. This could complicate debugging.	2022-01-23 13:55:48 +02:00
Aliaksandr Valialkin	233101137d	lib/blockcache: optimize blockcache a bit ... - Optimize Cache.RemoveBlocksFromPart(), so it doesn't need to iterate over all the cached blocks. - Cache blocks if there were no cache misses during the last 2 minutes. This may be the case when new blocks are added simultaneously to the storage and to the cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-23 13:08:55 +02:00
Aliaksandr Valialkin	9edf407144	lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin	4e05298756	lib/storage: properly limit cardinality when ingesting multiple samples for the same time series in a single request	2022-01-21 12:38:22 +02:00
Aliaksandr Valialkin	e3277918e4	lib/storage: verify that blocks in a single part are sorted by TSID when reading sequential blocks from the part ... This may help narrowing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:28 +02:00
Aliaksandr Valialkin	54ee71e16d	lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used. ... This may help nailing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:24 +02:00
Aliaksandr Valialkin	5159a9451f	lib/blockcache: add missing dependency after 145337792d ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:51:03 +02:00
Aliaksandr Valialkin	6ae584b9b3	lib/{mergeset,storage}: properly limit cache sizes for indexdb ... Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`, since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded, which could result to out of memory errors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin	da95516a1f	lib/promscrape: expose promscrape_stale_samples_created_total metric for monitoring the number of created stale samples	2022-01-14 01:00:40 +02:00
Aliaksandr Valialkin	dd91759f1f	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_node_provider_id label for discovered Kubernetes nodes in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/9603	2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin	bc18368c15	lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace ... See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881	2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin	de8299f465	lib/promscrape/discovery/dockerswarm: follow up after 68a117a25a ... - Document the bugfix at docs/CHANGELOG.md - Set __address__ field after copying commonLabels to the resulting map of discovered labels. This makes sure that the correct __address__ label is used.	2022-01-11 09:22:03 +02:00
Alexander Shtuchkin	45a92e6ce1	Fix for #2038: Make correct __address__ value for dockerswarm promscrape (#2041)	2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin	fa89f3e5a5	lib/promscrape: do not send staleness markers on graceful shutdown ... This follows Prometheus behavior. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079	2022-01-07 01:19:06 +02:00
Aliaksandr Valialkin	80fc3fda07	lib/storage: follow-up for 38bf5fc136	2022-01-05 16:02:17 +02:00
weng zhao	1e0fe615ad	vmstorage: fix query like {foo=~"bar|"} return extra timeseries cause by negative filter transformation malfunction (#2032) ... 1. L2749 make kb.B remain the value of comonPrefix instead of tf.prefix 2. L2762 avoid change tf.value from "bar|" to ".+r|"	2022-01-05 15:57:54 +02:00
Aliaksandr Valialkin	c1722003a2	lib/promscrape: scrape replicated targets at different offsets in vmagent replicated clustering mode ... This guarantees that the deduplication consistently leaves samples from the same vmagent replica. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets	2021-12-23 00:21:41 +02:00
Nikolay	6cdc934c3d	adds restore.lock (#1988) ... * adds restore.lock it must prevent from running storage after incomplete restore process https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 * return back flock file deletion * Apply suggestions from code review * wip * docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-22 13:10:56 +02:00
Aliaksandr Valialkin	727797a6fd	all: use logger.WithThrottler() where appropriate	2021-12-21 17:10:54 +02:00
Aliaksandr Valialkin	4dbf12254d	lib/promscrape: take into account the original job_name when creating an unique key per each scrape target ... This should handle the case when the original job_name has been changed in -promscrape.config , while the resulting job label remains the same because it is overriden via relabeling.	2021-12-21 16:42:42 +02:00
Roman Khavronenko	23e1de06ee	vmagent: add error log for skipped data block when rejected by receiv… (#1956) ... * vmagent: add error log for skipped data block when rejected by receiving side Previously, rejected data blocks were silently dropped - only metrics were update. From operational perspective, having an additional logging for such cases is preferable. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: throttle log messages about skipped blocks The new type of logger was added to logger pacakge. This new type supposed to control number of logged messages by time. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/logger: make LogThrottler public, so its methods can be inspected by external packages Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 16:42:38 +02:00
Aliaksandr Valialkin	053e85ff3d	all: typo fix: unexected -> unexpected	2021-12-20 17:40:13 +02:00
Aliaksandr Valialkin	406cb06f8c	lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase ... This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981	2021-12-20 17:26:07 +02:00
Aliaksandr Valialkin	f22aab411b	lib/storage: properly update per-part min_dedup_interval file contents after merge ... Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value This is a follow-up for 4ff647137a	2021-12-17 20:12:18 +02:00
Aliaksandr Valialkin	5bd4e47a9e	lib/promscrape: allow up to 5 redirects when scraping a target by default ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:45 +02:00
Aliaksandr Valialkin	d36fdbe537	lib/storage: deduplicate samples more thoroughly ... Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in higher disk space usage.	2021-12-15 16:00:30 +02:00
Aliaksandr Valialkin	bc3923111b	lib/storage: return dedup interval in milliseconds from GetDedupInterval() ... This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin	cdfe854c9b	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() ... This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:52:29 +02:00
Aliaksandr Valialkin	c922c7af9a	lib/storage: convert alternate regexps into Graphite wildcards inside __graphite__ pseudo-label ... For example, `{__graphite__=~"foo.(bar|baz)"}` is automatically converted to `{__graphite__=~"foo.{bar,baz}"}` before execution. This allows using multi-value Grafana template variables such as `{__graphite__=~"foo.($app)"}`.	2021-12-14 19:55:59 +02:00
Aliaksandr Valialkin	38f5bc7451	lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' ... This allows copy-pasting the url to http server from logs.	2021-12-08 16:15:12 +02:00
Aliaksandr Valialkin	9aa9b081a4	app/vminsert: add -maxLabelValueLen command-line flag ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908	2021-12-06 11:42:24 +02:00
Aliaksandr Valialkin	51f2eb3c46	lib/workingsetcache: fix unaligned 64-bit atomic operation panic on 32-bit architectures ... The panic has been introduced in 7275ebf91a	2021-12-03 01:22:30 +02:00
Aliaksandr Valialkin	d40441947a	app: allow specifying http and https urls in the following command-line flags ... * -promscrape.config * -relabelConfig * -remoteWrite.relabelConfig * -remoteWrite.urlRelabelConfig	2021-12-03 00:11:47 +02:00
Aliaksandr Valialkin	daaea1eb2c	app/vmauth: follow-up for 13368bed18 ... * Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md * Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file should be read from the given path. For example, in `-promscrape.config` at `vmagent`.	2021-12-02 23:34:15 +02:00
Aliaksandr Valialkin	b885a3b6e9	lib/httpserver: expose /-/healthy and /-/ready endpoints as Prometheus does ... This improves integration with third-party solutions, which rely on these endpoints. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833	2021-12-02 14:37:50 +02:00
Aliaksandr Valialkin	c540235470	app: use relative paths instead of absolute paths for the supported http handlers on the main page ... This allows hiding VictoriaMetrics components behind proxies, which serve pages at different path prefixes See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858	2021-12-02 13:54:15 +02:00
Aliaksandr Valialkin	d1289383eb	lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp	2021-12-02 13:44:01 +02:00
Aliaksandr Valialkin	37a2bea072	lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865	2021-12-02 13:33:50 +02:00
Aliaksandr Valialkin	2b7dee15dd	app/{vmbackup,vmrestore}: export internal metrics at /metrics http handler	2021-12-02 11:56:34 +02:00
Aliaksandr Valialkin	ab4be24397	app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches ... The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query: vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9	2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin	d4655beae8	lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics ... These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`	2021-12-02 09:13:20 +02:00
Aliaksandr Valialkin	2e43cd9d62	lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges	2021-12-01 12:30:03 +02:00
Nikolay	cf1d2f289b	removes FileSize from backup part key (#1872) ... * removes FileSize from backup part key it should fix download restoration for backups * Update lib/backup/common/part.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-01 12:30:03 +02:00
Aliaksandr Valialkin	71c0f7cce3	lib/storage: take into account -storage.minFreeDiskSpaceBytes when performing big merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-11-30 12:56:53 +02:00
guidao	6fa7ad69fc	fix #1830 (#1861) ... Co-authored-by: wangfeng <wangfeng@zhihu.com>	2021-11-30 01:16:15 +02:00
Aliaksandr Valialkin	975498d402	lib/protoparser/prometheus: follow-up for 8e338632a3 ... Do not spend CPU time on error message formatting if error logger is disabled	2021-11-30 00:51:15 +02:00
Nikolay	40f0726147	Changes unmarshallRow logger to noop for getRowsDiff (#1835)	2021-11-30 00:51:14 +02:00
Aliaksandr Valialkin	4ad397188e	lib/protoparser: do not log connection reset by peer error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections ... This error is expected, so there is no need in spamming the log with this error.	2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin	e93f46187d	lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent	2021-11-17 16:42:12 +02:00
Lan	6662714c6c	Add flag of S3ForcePathStyle (#1802)	2021-11-17 01:10:22 +02:00
Aliaksandr Valialkin	4fb19fe34b	all: consistently return application/json content-type without charset=utf-8 ... The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin	f41c02e475	lib/promscrape: improve logging for scrape_config_files parse errors ... Log the actual file path, which led to the parse error. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789	2021-11-08 13:34:26 +02:00
Aliaksandr Valialkin	847004fa77	app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin	bc72b83102	lib/promauth: do not show empty values in oauth2 config section at /config page	2021-11-05 12:54:10 +02:00
Aliaksandr Valialkin	d445d22c0c	lib/promscrape: add -promscrape.maxResponseHeadersSize command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:27:55 +02:00
Aliaksandr Valialkin	6873d6d893	lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp	2021-10-28 12:48:34 +03:00
Aliaksandr Valialkin	105deb164c	lib/logger: show only explicitly set command-line flags in logs ... This reduces initial verbosity in logs	2021-10-28 11:03:21 +03:00
Aliaksandr Valialkin	b626d6d606	lib/promscrape: add collapse and expand buttons per each group of targets from the same scrape job	2021-10-27 20:04:03 +03:00
Aliaksandr Valialkin	2ebee4e741	app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules ... Previously only the lower part of 64-bit hash was used for calculating the offset. This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash for calculating the offset.	2021-10-27 20:04:02 +03:00
Aliaksandr Valialkin	92d01db85a	lib/protoparser/prometheus: optimize GetRowsDiff() function ... This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 , since the provided profile shows that the majority of CPU and memory is spent in this function during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.	2021-10-27 18:55:25 +03:00
Aliaksandr Valialkin	16f1aaf0b5	lib/protoparser/prometheus: add a benchmark for GetRowsDiff	2021-10-27 18:55:23 +03:00
Aliaksandr Valialkin	99784b21c1	all: fix build issues and tests for Apple M1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:07:19 +03:00
Aliaksandr Valialkin	ad445a06cd	lib/promscrape: properly show proxy_url option value at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin	b08f51f5d3	lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set ... The response body isn't used if -promscrape.noStaleMarkers is set after the commit 2876137c92 , so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694	2021-10-22 16:49:21 +03:00
Aliaksandr Valialkin	6bc10f0623	lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed ... This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets This is a follow-up after 9882cda8b9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728	2021-10-22 13:55:39 +03:00
Aliaksandr Valialkin	a8bcc3c276	lib/promscrape: reduce memory usage if -promscrape.noStaleMarkers command-line flag is passed ... Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled. This should reduce memory usage when the scraped targets return large responses.	2021-10-22 13:22:08 +03:00
Nikolay	83e1dfccba	adds tab as second separator for graphite text protocol (#1733) ... * adds tab as second separator for graphite text protocol * changes indexFunc for indexAny * Update lib/protoparser/graphite/parser_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:29:27 +03:00
Aliaksandr Valialkin	d56e676d71	lib/flagutil: do not expose sensitive info (passwords, keys and urls) at /flags page	2021-10-20 00:51:15 +03:00
Aliaksandr Valialkin	5705f4b6d1	lib/httpserver: expose command-line flags at /flags page ... This should simplify debugging. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-20 00:46:54 +03:00
Aliaksandr Valialkin	a105b71116	lib/envflag: use flag.Set for setting the flags from env vars ... This should make visible the set flags at flag.Visit(), which is used later for logging and exporting the `is_set` label for these flags at /metrics page	2021-10-20 00:46:53 +03:00
Aliaksandr Valialkin	93511b4be7	lib/storage: log a warning when the -storageDataPath has less than -storage.minFreeDiskSpaceBytes ... This should improve the debuggability of the readonly feature. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727	2021-10-19 23:58:09 +03:00
Aliaksandr Valialkin	ea69eef375	lib/promscrape/discovery/kubernetes: log a warning if role: endpoints discovers more than 1000 targets per a single endpoint ... In this case `role: endpointslice` must be used instead. See the following references: * https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity * https://github.com/kubernetes/kubernetes/pull/99975 * https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398	2021-10-19 13:22:28 +03:00
Nikolay	e84a063209	changes job source for /target api (#1723) ... use jobNameOriginal instead of relabeled as prometheus does https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707	2021-10-19 09:00:05 +03:00
Aliaksandr Valialkin	fbcc8b5c7d	lib/promscrape: set honor_timestamps: true by default if this option isnt set explicitly in scrape configs ... This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-10-16 20:48:53 +03:00
Aliaksandr Valialkin	0a9be5ef9d	lib/promscrape: expose promscrape_series_limit_max_series and promscrape_series_limit_current_series metrics per each scrape target with the enabled unique series limiter	2021-10-16 19:14:13 +03:00
Aliaksandr Valialkin	99011c6b63	lib/promscrape: always initialize http client for stream parsing mode ... Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:19:48 +03:00
Aliaksandr Valialkin	0f4fda1bda	lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse ... This should reduce memory usage when scraping targets with big response bodies.	2021-10-16 13:00:11 +03:00
Aliaksandr Valialkin	0452a8d4e8	lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice ... This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode at the cost of the increased memory usage, since now the potentially big response is kept in the lastScrape byte slice per each scrapeWork. In practice the memory usage increase shouldn't be big, since the response size is usually much smaller than the parsed metrics from this response after the relabeling, which usually adds a big pile of target-specific labels per each metric.	2021-10-15 15:26:24 +03:00
Aliaksandr Valialkin	3e9beb0f8d	lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name ... This is a follow-up for 31b42b30b6	2021-10-15 12:27:31 +03:00
Aliaksandr Valialkin	25421fa2ae	lib/promscrape: add -promscrape.minResponseSizeForStreamParse command-line option for automatic switching to stream parsing mode when scraping targets with big responses ... This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes. This is common case in Kubernetes monitoring.	2021-10-14 12:30:55 +03:00
Aliaksandr Valialkin	bee130cc78	lib/promscrape: return error if sample_limit or series_limit options are set when stream parsing mode is enabled	2021-10-14 12:30:54 +03:00
Aliaksandr Valialkin	5b7d90d178	lib/promscrape: add ability to show the original labels for discovered targets at /targets page ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698	2021-10-13 16:44:34 +03:00
Roman Khavronenko	5dab25e8ad	lib/promscrape: make errcheck happy (#1703)	2021-10-13 15:11:45 +03:00
Aliaksandr Valialkin	c3a729d458	lib/promscrape: shard targets among cluster nodes after relabeling is applied ... This guarantees that targets with the same set of labels go to the same vmagent node. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495	2021-10-12 17:06:37 +03:00
Aliaksandr Valialkin	aeedfe2fe2	app/vmagent: expose -promscrape.config contents at /config page as Prometheus does ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-12 16:27:37 +03:00
Aliaksandr Valialkin	84aa08d93a	lib/promscrape: use Prometheus format for target labels at /targets page ... This should simplify copy-pasting the labels to/from PromQL / MetricsQL	2021-10-11 12:42:18 +03:00
Aliaksandr Valialkin	a7a1305395	lib/storage: fix unaligned access on 32-bit architectures. ... The bug has been introduced at a171916ef5	2021-10-08 19:38:20 +03:00
Aliaksandr Valialkin	a47754b689	lib/protoparser/clusternative: typo fix after 4fddcf4c83	2021-10-08 15:38:47 +03:00
Aliaksandr Valialkin	4fddcf4c83	app/{vminsert,vmstorage}: follow-up after a171916ef5 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-10-08 14:09:51 +03:00
Nikolay	a171916ef5	Adds read-only mode for vmstorage node (#1680) ... * adds read-only mode for vmstorage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269 * changes order a bit * moves isFreeDiskLimitReached var to storage struct renames functions to be consistent change protoparser api - with optional storage limit check for given openned storage * renames freeSpaceLimit to ReadOnly	2021-10-08 12:52:56 +03:00
Ziqi Zhao	1db3aeab36	fix some typos (#1678) ... Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>	2021-10-06 14:43:56 +03:00
Aliaksandr Valialkin	522a404b79	lib/promscrape: reduce memory allocations in mergeLabels() after 48e3e6c8df	2021-09-30 16:56:43 +03:00
Aliaksandr Valialkin	7b69d478ec	lib/protoparser: go fmt	2021-09-29 21:17:49 +03:00
Aliaksandr Valialkin	6167890d0e	lib/protoparser/prometheus: compare invalid Prometheus lines in full	2021-09-29 19:41:23 +03:00
Aliaksandr Valialkin	8dcf814c48	app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS ... The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls, while leaving support for `gcs://` urls for backwards compatibility.	2021-09-29 12:12:37 +03:00
Nikolay	9be5689b3f	changes auth validation for openstack (#1663) ... * changes auth validation for openstack must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 00:33:38 +03:00
Aliaksandr Valialkin	4e65bfcc00	app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API ... See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206	2021-09-29 00:12:26 +03:00
Aliaksandr Valialkin	d15d036a5a	lib/storage: properly handle {__name__=~"prefix(suffix1|suffix2)",other_label="..."} queries ... They were broken in the commit 00cbb099b6 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644	2021-09-23 21:52:31 +03:00
Aliaksandr Valialkin	d8de26bbfd	lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin	86bafe796c	lib/httpserver: add -enterprise and/or -cluster suffixes to short_version label of vm_app_version metric ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635	2021-09-21 23:12:50 +03:00
Aliaksandr Valialkin	c3e1f87048	lib/promrelabel: fix parsing regex: true in relabeling rules	2021-09-21 23:01:40 +03:00
Nikolay	dd53abf36d	changes protoparser apis for accepting reading from io.Reader (#1624) ... adds InsertHandlerForReader apis to vmagent	2021-09-20 14:54:20 +03:00
Nikolay	1ab2f844a2	makes filters optional for ec2 api requests (#1627) ... filters can be applied only for DescribeInstances requests, like prometheus does. related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin	1493461244	lib/storage: follow up after 00cbb099b6	2021-09-14 14:23:02 +03:00
faceair	61a51f7c15	lib/storage: optimize convert multiple values regexp filter to composite tag filter (#1610) ... * lib/storage: optimize convert multiple values regexp filter to composite tag filter * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-09-14 14:23:01 +03:00
Aliaksandr Valialkin	184e145570	docs: consistency renaming: Influx -> InfluxDB	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	b684624f67	lib/promscrape/discovery/docker: support host networking mode ... See https://github.com/prometheus/prometheus/issues/9116	2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin	6ed9f10da5	lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery ... See https://github.com/prometheus/prometheus/issues/8902	2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin	d90834da70	lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does ... See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin	279f37c9e7	lib/promscrape: make fmt	2021-09-12 13:35:21 +03:00
Aliaksandr Valialkin	6c97388dde	lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling ... See https://github.com/prometheus/prometheus/pull/8911	2021-09-12 13:35:20 +03:00
Aliaksandr Valialkin	09670479cd	lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric ... Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. The main idea is to perform the calculations on scrape_series_added and series_limit only if the set of metrics exposed by the target has been changed. Scrape targets rarely change the set of exposed metrics, so this optimization should reduce CPU usage in general case.	2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin	c339642858	lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric	2021-09-11 11:03:38 +03:00
Aliaksandr Valialkin	6d6cf1b6e0	lib/storage: verify that the tsidsFound contain the needed tsids in tests added at f4dead529f	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	5aaaa686a4	lib/promscrape: send stale markers for disappeared metrics like Prometheus does	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	c2f37f049b	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:12:53 +03:00
Aliaksandr Valialkin	d613a018c8	lib/promscrape: add ability to set series_limit and stream_parse options via relabeling ... This allows managing these options on a per-target basis. Typical use case: to manage these options for pods via Kubernetes annotations.	2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin	b64866e64c	lib/promscrape: add scrape_ prefix to job and target labels exported by promscrape_series_limit_rows_dropped_total metric ... This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.	2021-09-09 17:31:04 +03:00
Aliaksandr Valialkin	75c3514c5c	lib/promrelabel: add keep_metrics and drop_metrics actions to relabeling rules ... These actions simlify metrics filtering. For example, - action: keep_metrics regex: 'foo|bar|baz' would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted. The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above: - action: keep_metrics regex: - foo - bar - baz	2021-09-09 16:25:09 +03:00
mxlxm	42e07cfaea	reset deadline, fix #1562. (#1597) ... * reset deadline, fix #1562. reset deadline before we put it back to pool. * make errcheck happy	2021-09-07 20:54:17 +03:00
Aliaksandr Valialkin	c4df601f43	lib/promscrape: add the ability to limit the number of unique series per each scrape target ... The number of series per target can be limited with the following options: * Global limit with `-promscrape.maxSeriesPerTarget` command-line option. * Per-target limit with `max_series: N` option in `scrape_config` section. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561	2021-09-01 16:08:12 +03:00
Aliaksandr Valialkin	146c14d879	lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator ... This is a follow up commit after 31b42b30b6	2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin	18d7adf731	lib/protoparser/opentsdb: follow-up after 8ee75ca45a	2021-08-29 11:50:01 +03:00
envzhu	00dddfe02f	lib/protoparser/opentsdb: accept multiple spaces between fields in a row as a deliminator. (#1575)	2021-08-29 11:50:00 +03:00
Aliaksandr Valialkin	ca61d7c82b	lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus ... See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)	2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin	327034b54f	lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices ... This should fix service discovery for these roles in Kubernetes v1.22 and newer versions. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122 The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205	2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin	7fdb4db73d	lib/promscrape: add ability to load scrape configs from multiple files ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559	2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin	4a2d7aec7f	lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config	2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin	b885bd9b7d	lib/{mergeset,storage}: improve the detection of the needed free space for background merge ... This should prevent from possible out of disk space crashes during big merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560	2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin	67bc407747	lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets ... Store the scraped response body instead of storing the parsed and relabeld metrics. This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics. This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics. This should also reduce CPU usage, since the marshaling of the parsed and relabeld metrics has been substituted by response body copying. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin	c3b24882a7	lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape ... This will make timestamps for stale markers more consistent for timestamps for other samples	2021-08-19 14:19:54 +03:00
Aliaksandr Valialkin	8ee575dee9	lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does	2021-08-18 22:00:46 +03:00
Aliaksandr Valialkin	5d92fafc40	app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries ... This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin	f21fad53b4	lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag ... This option can be useful when vmagent consumes too much additional memory for staleness markers functionality and when staleness markers aren't needed.	2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin	db34c40aec	lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets ... This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509	2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin	5f13c519ee	lib/promscrape: restore red highlighting for DOWN targets at /targets page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461	2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin	c1f81f08d4	all: add support for Prometheus staleness markers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin	90efb5831b	lib/envflag: add a link to docs for -envflag.enable	2021-08-11 10:32:40 +03:00
Aliaksandr Valialkin	b877538622	app/vmagent: follow-up after fe445f753b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491	2021-08-05 09:51:00 +03:00
Omar Ghader	fe445f753b	feature: Add multitenant for vmagent (#1505) ... * feature: Add multitenant for vmagent * Minor fix * Fix rcs index out of range * Minor fix * Fix multi Init * Fix multi Init * Fix multi Init * Add default multi * Adjust naming * Add TenantInserted metrics * Add TenantInserted metrics * fix: remove unused metrics for vmagent * fix: remove unused metrics for vmagent Co-authored-by: mghader <marc.ghader@ubisoft.com> Co-authored-by: Sebastian YEPES <syepes@gmail.com>	2021-08-05 09:44:29 +03:00
Aliaksandr Valialkin	77bb9e1656	lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 ... See https://github.com/prometheus/prometheus/pull/8978	2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin	336a2aa2e0	lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does	2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin	c473d8ffe1	li/storage: re-use the per-day inverted index search code for searching in global index ... This allows removing a big pile of outdated code for global index search. This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486	2021-07-30 10:28:20 +03:00
Nikolay	6d47e750be	adds check for region with custom s3 endpoint (#1465)	2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin	1950f57316	lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-27 12:03:31 +03:00
Aliaksandr Valialkin	92628f9f07	lib/promrelabel: add tests for verifying that regex works as expected in single quotes and double quotes	2021-07-27 10:53:03 +03:00
Aliaksandr Valialkin	5d255846ac	all: add go:build lines for Go1.17 ... See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin	c857e05604	lib/promscrape: add missing whitespace at /targets page before up word	2021-07-26 12:23:06 +03:00
Aliaksandr Valialkin	376af3c956	lib/workingsetcache: switch from split cache to full cache after the cache size exceeds 95% of split capacity ... Previously the switch occurred when the cache size becomes 100% of its capacity. The cache size could never reach 100% capacity. This could prevent from switching from the split cache to full cache, thus reducing the cache effectiveness.	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	9d3f9da5ad	lib/storage: make sure the second call to DeduplicateSamples and deduplicateSamplesDuringMerge doesnt change samples	2021-07-15 12:18:38 +03:00
Aliaksandr Valialkin	e992754e79	lib/storage: remove cache directory if it contains reset_cache_on_startup file ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447	2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin	e6edb85fa2	lib/httpserver: add is_set label to flag metrics ... This label allows determining the set flags with the query `flag{is_set="true"}`	2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin	51cd19d2e3	lib/storage: reset perKeyMisses stats less frequently ... This should reduce CPU usage for queries executed with intervals higher than 30 seconds	2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin	3f705fe8d7	lib/storage: properly limit the size of storage/date_metricID cache	2021-07-12 14:25:28 +03:00
Aliaksandr Valialkin	ef3c58d7a3	lib/storage: properly determine when the deduplication is needed in needsDedup ... Previously needsDedup() could return true if the de-duplication wasn't needed for the following case: d < interval / \ | v | v | interval interval Now it properly returns false for this case	2021-07-12 10:54:51 +03:00
Aliaksandr Valialkin	41754e12f8	lib/mergeset: cache indexBlock items only on the second request ... This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.	2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin	ceda2b1df4	lib/httpserver: print full requestURI in httpserver.Errorf ... This should simplify debugging.	2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin	9826f7c1be	lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) ... This should reduce the cache size and memory usage for the indexdb/dataBlocks cache	2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin	74ace9340d	lib/storage: periodically reset prefetchedMetricIDs cache in order to limit its size under high churn rate	2021-07-07 10:59:39 +03:00
Aliaksandr Valialkin	a846febc89	Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" ... This reverts commit 7c6d3981bf. Reason for revert: high contention at bucket16Pool on systems with big number of CPU cores. This slows down query processing significantly.	2021-07-06 18:26:56 +03:00
Aliaksandr Valialkin	b805a675f3	lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects ... This should reduce memory usage on systems with big number of CPU cores, since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains a separate pool inmemoryPart objects per each CPU core. Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized by big sizes of inmemoryPart objects.	2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin	d8e7c1ef27	lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method ... This reduces the load on memory allocator in Go runtime in production workload.	2021-07-06 16:33:24 +03:00
Aliaksandr Valialkin	db6bd69475	lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload ... CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small. This results in the increased load on memory allocation code in Go runtime. Increase the pool capacity in order to reduce the load on Go runtime.	2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin	fd32855a6c	lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds ... This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered (aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.	2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin	22c6e64bbc	lib/storage: consistency renaming: tagCache -> tagFiltersCache ... This improves code readability	2021-07-06 11:03:30 +03:00
Aliaksandr Valialkin	21abf487c3	lib/workingsetcache: properly update stats for requests and cache misses ... Previously the stats for cache misses could be improperly counted, because it had inflated cache misses if the entry was missing in the curr cache, but was existing in the prev cache. The same applies to cache requests - they were inflated if the entry was missing in the curr cache.	2021-07-06 10:54:38 +03:00
Aliaksandr Valialkin	e5031d9aee	lib/workingsetcache: fix cache capacity calculations after 4f0003f182	2021-07-05 17:16:35 +03:00
Aliaksandr Valialkin	bd71f102e8	lib/workingsetcache: typo fixes after d0c830039d	2021-07-05 15:35:51 +03:00
Aliaksandr Valialkin	4b25e627f8	lib/workingsetcache: properly switch to whole mode ... Previously the switch from `split` to `whole` mode had been performed too early, e.g. when the current cache size became bigger than 1/4 of the allowed cache size. Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size. This change can reduce memory usage for data ingestion path when big number of active time series are ingested.	2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin	51516b96e6	lib/storage: tune cache sizes according to production workload	2021-07-05 15:14:45 +03:00
Aliaksandr Valialkin	f12f97daa1	lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes ... One minute cache timeout result in slower queries in some production workloads where the interval between query execution is in the range 1 minute - 2 minutes.	2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin	377bb06b47	lib/cgroup: set GOGC to 50 by default if it isn't set ... This should reduce memory usage for typical VictoriaMetrics workloads by up to 50%	2021-07-05 12:34:01 +03:00
Aliaksandr Valialkin	8055439fe4	lib/storage: properly detect free disk space shortage during data merge ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin	6fc3696260	lib/promscrape/discovery/consul: use case-insensitive comparison for service names ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424	2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin	61e483a01c	lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime ... This is a follow-up for b84aea1e6e	2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin	72de54f93e	lib/promauth: cache the client TLS certificate for up to a second ... This should reduce CPU usage when TLS connections are established at a high rate.	2021-07-02 13:20:18 +03:00
Aliaksandr Valialkin	1c12c0f79c	lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does ... This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2021-07-01 15:43:43 +03:00
Nikolay	6bd2309449	fixes /targets button style (#1423) ... * fixes /targets button style https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422 * updates boostrap version	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	71c856beb8	lib/workingsetcache: reset the cache mode when the cache is reset ... This should reduce memory usage if the working set is reduced after the cache reset.	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	bced9ee666	lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute ... This should reduce memory usage on a system with high number of active time series and a high churn rate. One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).	2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin	b7c0b3dde3	lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs ... This should reduce memory usage for the pool on systems with big number of CPU cores. The sync.Pool maintains per-CPU pools, so the total number of objects in the pool is proportional to the number of available CPU cores. The channel limits the number of pooled objects by its own capacity. This means smaller number of pooled objects on average.	2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin	2edfea8c36	lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID	2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin	609ad6d9bf	lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache ... This should prevent from stats overwriting when the previous indexdb is queried.	2021-06-29 13:11:32 +03:00
Aliaksandr Valialkin	0c4c630839	lib/promscrape: typo fix in /targets output ... The typo has been introduced in fb72a2133f Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408	2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin	97d1ccfc8e	lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common ... This is a follow up after c85a5b7fcb	2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin	4461e20e7d	lib/promscrape: consistently sort service discovery routines ... This should simplify further maintenance of the code	2021-06-25 13:22:16 +03:00
Lu Jiajing	12b4cbb68f	Support Docker ServiceDiscovery (#1402) ... * add docker discovery * add test * add labels test and add scrape work * remove TODO * refactor to merge apiConfig and sdConfig * apply suggestion	2021-06-25 13:22:16 +03:00
Nikolay	501429c3ff	adds missing MustStop call to do and http sd (#1404)	2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin	b84aea1e6e	lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) ... This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores	2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin	a22f37599b	lib/storage: tune tag filters search logic ... Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624 The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats. This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters for further queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin	f10fa0d1d7	lib/promscrape/discovery/consul: properly pass namespace to Consul watcher ... Follow-up for 58a2989fe7	2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin	4adf6c9766	lib/promscrape/discovery/http: follow up after e307bbb29a	2021-06-22 13:42:10 +03:00
Nikolay	e03a3d3a36	adds http_sd (#1399) ... * adds http_sd * adds X-Prometheus-Refresh-Interval-Seconds header * Update lib/promscrape/discovery/http/api.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin	3ab3902f17	lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does	2021-06-22 13:18:51 +03:00
Nikolay	827a2396d2	adds consul enterprise namespace support (#1400) ... * adds consul enterprise namespace support * Update lib/promscrape/discovery/consul/consul.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin	f9069ba32a	lib/promscrape: show jobs with empty scrape targets on /targets page	2021-06-18 10:54:12 +03:00
Nikolay	9ea1dca3dd	fixes DO service discovery labels (#1389) ... adds test for digitalocean sd	2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin	a207be3ffb	lib/storage: fix infinite loop introduced in aa9b56a046 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin	0efd37cec1	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores ... VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit 56b6b893ce . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin	b133de1e37	lib/storage: move deletedMetricIDs set from indexDB to Storage ... This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB). This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation. See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 . Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart. This should be OK in most cases.	2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin	ebaf68bcb0	lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error ... This is a follow-up for af90c3c43b	2021-06-14 15:20:38 +03:00
faceair	2ea187e801	lib/protoparser: stop read when callback error (#1380)	2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin	5f91a701fa	lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377	2021-06-14 14:04:35 +03:00
Nikolay	e42da47608	adds digital ocean sd (#1376) ... * adds digital ocean sd config * adds digital ocean sd https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367 * typo fix	2021-06-14 13:19:29 +03:00
Aliaksandr Valialkin	df057177a0	lib/promscrape: increase the duration for reading the full response in stream parsing mode ... Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin	074b11fa69	lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin	87d221f78a	lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error ... This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin	0672cfffa2	app/vmauth: properly handle http.ErrAbortHandler panic ... This panic can be raised by the reverseProxy on aborted request to the backend. So handle it (e.g. suppress) at reverseProxy.ServeHTTP call. Do not suppress the panic at lib/httpserver generic HTTP handler, since it may result in an inconsistent state left after the panicking handler. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353	2021-06-11 12:54:37 +03:00
Aliaksandr Valialkin	ce10bdc82a	lib/storage: reset cache on disk during series deletion and during indexdb rotation ... This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347	2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin	eb335d2c29	lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard	2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin	d06c0e7a94	lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores	2021-06-11 10:49:02 +03:00
Nikolay	2c1611d316	disables panic for net/httpAbortHandler (#1355)	2021-06-09 12:12:45 +03:00
Aliaksandr Valialkin	1e4a64844d	lib/storage: properly account the number of loops spent when matching for or suffixes ... This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin	e7d353ee6a	lib/promrelabel: add tests for labelsToString() function	2021-06-04 20:42:14 +03:00
Aliaksandr Valialkin	269e35d676	app/{vmagent,vminsert}: follow-up after 2fe045e2a4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343	2021-06-04 20:33:22 +03:00
jelmd	d8b46908db	new feature: debug relabeling (#1344) ... * new feature: relabel logging Use scrape_configs[x].relabel_debug = true to log metric names inkl. labels before and after relabeling. After relabeling related metrics get dropped, i.e. not submitted to servers. * vminsert wants relabel logging, too.	2021-06-04 20:33:21 +03:00
Nikolay	3d89c01d07	fixes solaris build (#1345)	2021-06-04 11:56:06 +03:00
Hason Chan	439c2ed510	fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350)	2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin	fc2565b4ee	lib/storage: reduce memory allocations when syncing dateMetricIDCache	2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin	0b9f0de0a1	lib/promscrape: fix tests after f0c21b6300	2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin	6865f3b497	Revert "lib/mergeset: remove a pool for inmemoryBlock structs" ... This reverts commit 793fe39921. Reason to revert: production testing revealed possible slowdown when registering big number of new time series	2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin	7b33bc67a1	lib/mergeset: remove a pool for inmemoryBlock structs ... The pool for inmemoryBlock struct doesn't give any performance gains in production workloads, while it may result in excess memory usage for inmemoryBlock structs inside the pool during background merge of indexdb.	2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin	97de72054e	docs: document f0c21b6300	2021-05-27 15:04:13 +03:00
faceair	b801b299f0	lib/promscrape: apply body size & sample limit to stream parse (#1331) ... * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 15:04:11 +03:00
Aliaksandr Valialkin	49490ae5a7	lib/protoparser/clusternative: remove duplicate cannot read packet size phrase from the log message ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	c85084b659	lib/handshake: pass io.EOF unmodified to the caller for BufferedConn.Read, so it could properly detect the end of stream	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	10b2855949	lib/storage: fix spelling typo: borken->broken ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	6b90570ed3	lib/uint64set: store pointers to bucket16 instead of bucket16 objects in bucket32 ... This speeds up bucket32.addBucketAtPos() when bucket32.buckets contains big number of items, since the copying of bucket16 pointers is much faster than the copying of bucket16 objects. This is a cpu profile for copying bucket16 objects: 10ms 13.43s (flat, cum) 32.01% of Total 10ms 120ms 650: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 651: b.b16his[pos] = hi . 13.31s 652: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . . 653: b16 := &b.buckets[pos] . . 654: *b16 = bucket16{} . . 655: return b16 . . 656:} This is a cpu profile for copying pointers to bucket16: 10ms 1.14s (flat, cum) 2.19% of Total . 100ms 647: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 648: b.b16his[pos] = hi 10ms 700ms 649: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . 330ms 650: b16 := &bucket16{} . . 651: b.buckets[pos] = b16 . . 652: return b16 . . 653:}	2021-05-25 14:27:52 +03:00
Aliaksandr Valialkin	1c16cbacf5	lib/storage: do not stop data ingestion on the first error in Storage.AddRows ... Continue data ingestion for the rest of blocks.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	2601844de3	lib/storage: limit the number of rows per each block in Storage.AddRows() ... This should reduce memory usage when ingesting big blocks or rows.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	95b735a883	lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows ... This should reduce memory usage a bit on data ingestion path	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	0f84503880	lib/bloomfilter: fix TestLimiterConcurrent	2021-05-24 05:18:29 +03:00
Aliaksandr Valialkin	745eda9e87	lib/fs: do not pass done callback to tryRemoveAll() func ... This improves code readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-24 05:00:53 +03:00
Aliaksandr Valialkin	402a8ca710	lib/storage: do not populate MetricID->MetricName cache during data ingestion ... This cache isn't needed during data ingestion, so there is no need in spending RAM on it. This reduces RAM usage on data ingestion path by 30%	2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin	0fc857d363	lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds ... Log these messages if the merge takes more than 30 seconds instead of 10 seconds.	2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin	71ff7ee18d	lib/promauth: follow-up after 5b8176c68e	2021-05-22 18:02:03 +03:00
Nikolay	2780d6dbcd	basic OAuth2 support for remoteWrite and scrape targets (#1316) ... * adds OAuth2 support for remoteWrite and scrapping * adds tests changes init	2021-05-22 18:02:01 +03:00
Aliaksandr Valialkin	89e1a45cdb	lib/fs: concurrently remove up to 1024 blocked NFS directories ... Previously the blocked directories were removed sequentially by a single goroutine. This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second, when big number of LSM parts are created and removed at high rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:58:08 +03:00
Aliaksandr Valialkin	23355ca34c	lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full ... This should reduce the probability of the panic on a highly loaded VictoriaMetrics accepting millions of samples per second. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:21:35 +03:00
Aliaksandr Valialkin	d77db9d813	all: do not skip SIGHUP signal during service initialization ... This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin	69e365cd48	Makefile: update golangci-lint from v1.29.0 to v1.40.1	2021-05-20 18:30:24 +03:00
Aliaksandr Valialkin	da0b32c31a	app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour ... These numbers are exposed via the following metrics: - vmagent_hourly_series_limit_current_series - vmagent_daily_series_limit_current_series Expose also the limits via the following metrics: - vmagent_hourly_series_limit_max_series - vmagent_daily_series_limit_max_series	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	165a9f9200	app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	7aad5c3f76	app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	110a888e39	lib/promscrape/discovery/kubernetes: make golangci-lint happy by removing empty branches	2021-05-20 12:00:17 +03:00
Aliaksandr Valialkin	e228f479a5	lib/storage: remove possible data race when logging dropped labels	2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin	9d97f44772	lib/promscrape/discovery/kubernetes: reload objects on object parse error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 23:27:24 +03:00
Aliaksandr Valialkin	74ef40034c	lib/httpserver: typo fix in -http.shutdownDelay command-line flag description: servier -> server	2021-05-18 16:25:27 +03:00
Aliaksandr Valialkin	c507faec0b	lib/promscrape/discovery/kubernetes: simplify the reload logic for urlWatcher.objectsByKey	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	0f54c0121b	lib/promscrape/discovery/kubernetes: properly update vm_promscrape_discovery_kubernetes_scrape_works metric ... Previously it wasn't descreased during config update.	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	9f62d348db	lib/promscrape/discovery/kubernetes: log errors and stop service discovery when unexpected updates are received from Kubernetes API server ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	6ea191d196	docs: dealay -> delay	2021-05-18 01:07:32 +03:00
Aliaksandr Valialkin	c4ed50ae54	lib/promrelabel: add tests for conditional removal of label on another label match ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1294	2021-05-18 00:23:23 +03:00
Aliaksandr Valialkin	8764b0ae21	lib/promscrape/discovery/kubernetes: key ScrapeWork objects by urlWatcher instead of namespace ... This makes the code less fragile if urlWatcher would depend on additional to namepsace properties. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-17 23:49:48 +03:00
Aliaksandr Valialkin	e08287f017	lib/promscrape: reload auth tokens from files every second ... Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:03:35 +03:00
Aliaksandr Valialkin	a6cb4f10a7	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport ... By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin	e3f61d540b	lib/promscrape: limit scrape_timeout by scrape_interval like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1281	2021-05-13 16:10:42 +03:00
匠心零度	d5285ecaf0	fix vagent imbalance problem (#1292) ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-05-13 11:19:30 +03:00
Aliaksandr Valialkin	f13585dc5d	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:09 +03:00
Aliaksandr Valialkin	d13906bf1f	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services ... This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:07 +03:00
Aliaksandr Valialkin	66c6976723	lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for b50024812e	2021-05-13 09:27:35 +03:00
Nikolay	8743bf541f	adds cgroupsv2 support (#1283) ... * adds cgroupv2 limits support https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269 * small fix * changes Atoi to ParseUint	2021-05-13 09:27:33 +03:00
Aliaksandr Valialkin	2839055513	lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss	2021-05-13 09:01:05 +03:00
Aliaksandr Valialkin	008ae25b3a	lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate ... These functions are non-trivial, while their code has minimal differences. It is better from maintainability PoV to merge these functions into a single function.	2021-05-12 18:01:08 +03:00
Nikolay	be87be34a4	Adds tsdb match filters (#1282) ... * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin	027607db3e	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices scrape targets every 5 seconds, since they may depend on changed service and pod objects ... This should make endpoints and endpointslices scrape targets eventually consistent with the maximum delay of 5 seconds after the related service or pod object changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-12 14:12:43 +03:00
Aliaksandr Valialkin	1d32b008c6	lib/httpserver: add new X-Server-Hostname header instead of overwriting already exsiting header ... This makes possible tracking origins of chained requests over multiple hops.	2021-05-11 23:47:19 +03:00
Aliaksandr Valialkin	f1317f7c6c	lib/httpserver: return X-Server-Hostname http header in all the responses for better debuggability	2021-05-11 22:04:41 +03:00
Aliaksandr Valialkin	4e59cf4380	lib/storage: properly apply time range when matching an empty filter ... It must match all the time series on the given time range. Previously it was matched to all the time series without the restriction on the given time range.	2021-05-11 01:09:35 +03:00
Aliaksandr Valialkin	326cf83eb4	lib/storage: remove dead code after the commit 3ccf7ea20c	2021-05-08 20:15:59 +03:00
Aliaksandr Valialkin	9c505d27dd	lib/ingestserver: properly close incoming connections during graceful shutdown	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	4a5f45c77e	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	e6c19cb09d	lib/promscrape/discovery/kubernetes: start watchers for pods and services before starting watchers for endpoints ... This should eliminate possible race when an update on endpoints depends on pods and/or services, which are missing in the cache yet. This could result in missing targets based on endpoints or endpointslices. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-05 12:23:16 +03:00
Aliaksandr Valialkin	43c52ff77a	lib/storage: use WARNING instead of INFO level for logging dropped labels	2021-05-03 13:57:28 +03:00
Aliaksandr Valialkin	ec6becd3f5	lib/httpserver: stop the process on panics in request handlers ... Panics may leave the process in inconsistent state. That's why it is better to stop the process after the panic instead of recovering from the panic. Unfortunately, the standard net/http.Server recovers panics in request handlers. See https://github.com/golang/go/issues/16542 . That's lib/httpserver must stop the process on itself after the panic.	2021-05-03 12:00:44 +03:00
Nikolay	62d58324dd	adds stalePartsRemover (#1261) ... for new created partitions	2021-05-03 11:34:33 +03:00
Aliaksandr Valialkin	60ffbcbb99	lib/promrelabel: add tests for removing the specified {label="value"} pair	2021-05-03 11:26:58 +03:00
Aliaksandr Valialkin	b43ba6d85f	lib/storage: log dropped labels if the number of labels in a metric exceeds -maxLabelsPerTimeseries command-line flag value ... This should improve debuggability for this case.	2021-05-01 09:29:56 +03:00
Aliaksandr Valialkin	8be1cb297b	app/vmagent: list user-visible endpoints at http://vmagent:8429/ ... While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics	2021-04-30 09:38:23 +03:00
Aliaksandr Valialkin	421a92983a	lib/promscrape/discovery/kubernetes: remove a mutex at urlWatcher - use groupWatcher mutex for accessing all the urlWatcher children ... This simplifies the code a bit and reduces the probability of improper mutex handling and deadlocks.	2021-04-29 10:17:45 +03:00
Nikolay	535b3ff618	vmagent kubernetes_sd tests (#1253) ... * first part of tests for kubernetes sd * makes linter happy * added more test cases * adds pub/sub for tests	2021-04-29 10:17:45 +03:00
Aliaksandr Valialkin	e37e1b1e34	lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures ... The panic has been introduced in 56b6b893ce	2021-04-27 16:42:19 +03:00
Aliaksandr Valialkin	2d1d60118d	lib/mergeset: split rows ingestion among multiple shards ... This improves rows ingestion on systems with many CPU cores by reducing lock contention. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244 Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243	2021-04-27 15:45:11 +03:00
Aliaksandr Valialkin	b3da457629	lib/promscrape/discovery/kubernetes: fix a deadlock introduced in eddba29664 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 Thanks to @f41gh7 for providing the initial idea for deadlock fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1248	2021-04-27 14:59:56 +03:00
Aliaksandr Valialkin	cba2d13456	lib/storage: typo fix in info message when deleting the part outside the configured retention ... Previously the message was displaying incorrect retention time	2021-04-27 13:33:36 +03:00
Aliaksandr Valialkin	f14412321b	lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value	2021-04-27 00:26:32 +03:00
Aliaksandr Valialkin	320983f650	lib/promscrape: apply scrape_timeout on receiving the first response byte for stream_parse: true scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047	2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin	34321e5f8d	lib/promscrape/discovery/kubernetes: refresh role: endpoints targets on service object removal as Prometheus does ... This is a follow-up for ae37cfd528 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:27:29 +03:00
Aliaksandr Valialkin	db27dbab5e	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices targets on service object update like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:12:22 +03:00
Aliaksandr Valialkin	ab8008d6d7	lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142	2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin	6dc5d3b357	all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com	2021-04-20 20:20:01 +03:00
Aliaksandr Valialkin	1088113110	lib/uint64set: remove memory allocation in getOrCreateSmallPool() ... This partially reverts fb82c4b9fa It has been appeared that the additional memory allocation may result in higher GC pauses. It is better to spend CPU time on copying bigger bucket16 structs instead of increasing query latencies due to higher GC pauses	2021-04-14 12:31:53 +03:00
Aliaksandr Valialkin	72c41323fa	lib/storage: code clarification: remove caching the found metricName in searchMetricName	2021-04-13 10:20:35 +03:00
Artem Navoiev	c3dcfdef8c	improve docs for cli flags (#1202) ... * improve docs for cli flags * improve docs for cli flags.2	2021-04-12 12:28:36 +03:00
Aliaksandr Valialkin	3b1f0cb3f6	lib/promscrape: create a single swosFunc per scrape_config	2021-04-08 09:31:38 +03:00
Aliaksandr Valialkin	276dbc2133	lib/promscrape: do not spend CPU time on constructing scrapeWork key if clustering is disabled	2021-04-07 21:55:06 +03:00
Aliaksandr Valialkin	59ccc43e3a	lib/storage: properly handle big time ranges passed to /api/v1/labels and /api/v1/label/<labelName>/values ... It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch	2021-04-07 13:33:10 +03:00
Aliaksandr Valialkin	02b83e0957	lib/promscrape/discovery: remove superflouos check in registerPendingAPIWatchers ... The check `_, ok := uw.aws[aw]; !ok` isn't needed, since aw cannot exist in uw.aws because of the check inside subscribeAPIWatcher	2021-04-07 13:10:04 +03:00
Aliaksandr Valialkin	db56ee0e28	lib/promscrape/discovery/kubernetes: register pending apiWatchers in uw.aws	2021-04-06 11:11:53 +03:00
Aliaksandr Valialkin	edd66b7e82	lib/promscrape/discovery/kubernetes: remove superflouos mustStart and mustStop functions	2021-04-05 22:43:49 +03:00
Lu Jiajing	4ee6def68b	fix access to nil *url.URL (#1180) ... * fix access to nil *url.URL Signed-off-by: Megrez Lu <lujiajing1126@gmail.com> * Update lib/promscrape/discovery/kubernetes/api_watcher.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-05 22:26:43 +03:00
Aliaksandr Valialkin	7eca60694e	lib/promscrape/discovery/kubernetes: reduce CPU time spent on registering big number of Kubernetes objects shared among big number of scrape jobs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 22:05:02 +03:00
Aliaksandr Valialkin	9da2ef3d8f	lib/promscrape/discovery/kubernetes: load objects missing in local cache from api seriver in getObjectByRole() ... This should fix possible race for `role: endpoints` and `role: endpointslices` service discovery, when the referred `pod` and `service` objects aren't propagated to urlWatcher cache yet. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182#issuecomment-813353359 for details.	2021-04-05 20:31:22 +03:00
Aliaksandr Valialkin	4d6a3aba4d	lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030	2021-04-05 19:26:51 +03:00
Aliaksandr Valialkin	fe084fdd33	lib/promscrape/discovery/kubernetes: synchronously load Kubernetes objects on first access ... Remove async registration of apiWatchers, since it breaks discovering `role: endpoints` and `role: endpointslices` targets, which depend on pod and service objects. There is no need in reloading `endpoints` and `endpointslices` targets if the referenced `pod` or `service` objects change, since in this case the corresponding `endpoints` and `endpointslices` objects should also change because they contain ResourceVersion of the referenced `pod` or `service` objects, which is modified on object update. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 14:37:07 +03:00
Aliaksandr Valialkin	c8b7c32e23	lib/proxy: typo fix after a5c5b54c22	2021-04-05 14:37:06 +03:00
Aliaksandr Valialkin	e0b687171b	lib/proxy: add support for socks5 over tls proxy	2021-04-05 13:00:42 +03:00
Aliaksandr Valialkin	90da332ea0	lib/promscrape: pass X-Prometheus-Scrape-Timeout-Seconds header to scrape targets as Prometheus does	2021-04-05 12:15:14 +03:00
Nikolay	d0b664454b	adds socks5 support for fasthttp client (#1178) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1177 Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-04 01:43:59 +03:00
Aliaksandr Valialkin	dd19fab7c9	lib/promscrape: properly send full url in GET request via simple HTTP proxy ... This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 01:20:43 +03:00
Aliaksandr Valialkin	ab9e1eb41f	lib/promscrape: support for simple HTTP proxies without CONNECT method support such as https://github.com/prometheus-community/PushProx ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 00:40:58 +03:00
Aliaksandr Valialkin	27d8a5d2c0	lib/promscrape: add tests for authorization config, which has been added in df148f48b7	2021-04-03 22:14:03 +03:00
Aliaksandr Valialkin	ae5c20a34c	lib/proxy: log response body on non-200 response code ... This should improve debuggability for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-03 03:03:41 +03:00
Aliaksandr Valialkin	87700f1259	lib/promscrape: add support for authorization config in -promscrape.config as Prometheus 2.26 does ... See https://github.com/prometheus/prometheus/pull/8512	2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin	2825a1e86d	lib/promscrape: add follow_redirect option to scrape_configs section like Prometheus does ... See https://github.com/prometheus/prometheus/pull/8546	2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin	245eba8896	lib/promscrape/discovery/kubernetes: properly track objects with the same names in multiple namespaces ... This is a follow-up for 12e4785fe8 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:46:34 +03:00
Aliaksandr Valialkin	eee860f83d	lib/promscrape/discovery/kubernetes: properly discover targets in multiple namespaces ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:29:24 +03:00
Nikolay	00157f3ec6	Adds aws ECS credentials support (#1175)	2021-04-02 13:11:03 +03:00
Aliaksandr Valialkin	512addc608	app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage ... This option can be useful when samples for the same time series are ingested with distinct order of labels. For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.	2021-03-31 23:27:21 +03:00
Aliaksandr Valialkin	ae1c653d55	lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels	2021-03-31 21:22:40 +03:00
Aliaksandr Valialkin	887e67f13b	lib/uint64set: improve Set.Has() performance scalability on multi-CPU system ... Do not update bucket32.hint on Set.Has() call, since it leads to memory ping-pong between CPU cores multi-CPU system	2021-03-29 12:34:19 +03:00
Aliaksandr Valialkin	940a547116	lib/storage: do not update b.nextIdx if no samples are removed because of retention	2021-03-29 12:13:38 +03:00
Aliaksandr Valialkin	7d87d42a91	lib/promscrape/discovery/kubernetes: typo fix in error message	2021-03-26 12:46:33 +02:00
Aliaksandr Valialkin	a920e71809	lib/promscrape/discovery/kubernetes: properly handle too old resource version error message from Kubernetes watch API	2021-03-26 12:28:35 +02:00
Aliaksandr Valialkin	9c2be144cf	app/vmselect: log the metric which trigger rollup result cache reset ... This should help finding the source of stale metrics	2021-03-25 21:32:28 +02:00
Aliaksandr Valialkin	f971fe86cd	lib/storage: tune loopsCountPerMetricNameMatch according to production workload	2021-03-25 13:48:17 +02:00
Aliaksandr Valialkin	6b1f807418	app/vmagent: add -promscrape.consul.waitTime command-line flag for configuring Consul service discovery wait time ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144	2021-03-23 19:34:12 +02:00
Aliaksandr Valialkin	9947c65df3	lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock ... This should speed up Search.NextMetricBlock a bit	2021-03-23 17:59:34 +02:00
Nikolay	19a40faf8e	changes consul_service label value (#1143) ... according to prometheus discovery. It should mitigate issue with case sensetive services https://github.com/hashicorp/consul/issues/5707	2021-03-23 15:37:06 +02:00
Aliaksandr Valialkin	12ca0efc19	lib/storage: respect the deadline passed to Storage.SearchMetricNames	2021-03-22 23:03:00 +02:00
Aliaksandr Valialkin	40e47935e7	lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache	2021-03-22 23:02:59 +02:00
Aliaksandr Valialkin	1618b0ca6d	lib/storage: tune loopsCountPerMetricNameMatch	2021-03-22 12:57:54 +02:00
Aliaksandr Valialkin	7503111feb	lib/storage: small code simplification after 6cee5338b2	2021-03-18 15:22:39 +02:00
Aliaksandr Valialkin	4443254fb9	lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars ... The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137	2021-03-18 14:57:39 +02:00
Aliaksandr Valialkin	e03233f441	lib/fs: reduce the frequency of failed to remove directory ... due to NFS lock log warnings ... Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.	2021-03-18 13:23:43 +02:00
Aliaksandr Valialkin	b859fe7879	lib/storage: faster move heavy filters to the end of list	2021-03-17 15:11:56 +02:00
Aliaksandr Valialkin	5e77a939c2	all: make go vet happy	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	41fe707bec	lib/storage: limit loops count in order to reduce max CPU usage during filter search	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	e2a0c8bd72	lib/storage: do not modify filterLoopsCount stats with loopsCount stats ... Such a modification can result in incorrect filter sorting later	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	b997f4a418	all: make golangci-lint happy after the commit 6378205415	2021-03-17 00:24:31 +02:00
Aliaksandr Valialkin	8005ba26b9	lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics ... This is a follow-up for 18cfc4be7b See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:19:30 +02:00
Nikolay	b293f81eb8	Adds udp6 support for ingest servers (#1134) ... with flag -enableUDP6 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:19:29 +02:00
Aliaksandr Valialkin	1cfe3f872f	lib/uint64set: optimize bucket16.addMulti a bit	2021-03-16 21:08:38 +02:00
Aliaksandr Valialkin	727ded9d4e	lib/storage: time series search optimization according to production workload profiling ... Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down the function by multiple times when it finds big number of metricIDs (tens of millions).	2021-03-16 20:08:43 +02:00
Aliaksandr Valialkin	f4a44d6c0d	lib/storage: further tuning for time series search	2021-03-16 18:47:29 +02:00
Aliaksandr Valialkin	d074326970	app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate	2021-03-15 22:40:28 +02:00
Aliaksandr Valialkin	c4d0c20de2	lib/influxutils: return response compatible with InfluxDB 1.8.4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:20:38 +02:00
f41gh7	b7c9f3676e	typo fix	2021-03-15 22:20:37 +02:00
Aliaksandr Valialkin	e2717d84c0	all: various fixes in command-line flag descriptions	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	776b8b32ca	app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	fc902734d9	lib/storage: further tuning for time series selector code	2021-03-15 20:32:37 +02:00
Aliaksandr Valialkin	dc70e7de76	lib/uint64set: reduce the size of bucket16 by storing smallPool by pointer. ... This reduces CPU time spent on bucket16 copying inside bucket13.addBucketAtPos.	2021-03-15 17:27:03 +02:00
Aliaksandr Valialkin	cc6b1a5941	lib/uint64set: optimize Set.AddMulti for large sorted sets	2021-03-15 17:10:54 +02:00
Aliaksandr Valialkin	f023e92f1a	lib/uint64set: optimize bucket16.add and bucket16.addMulti a bit	2021-03-15 16:58:24 +02:00
Aliaksandr Valialkin	1c26020080	lib/storage: tune per-day index search	2021-03-15 13:36:36 +02:00
Aliaksandr Valialkin	7f52aae20c	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics ... Do not cache too big byte buffers and too big writeRequestCtx objects, since it is cheaper to re-create them instead of wasting RAM for their caching. This reverts 7f6f350ee1	2021-03-15 11:49:29 +02:00
Aliaksandr Valialkin	33cd6c26d3	lib/promscrape: return back the logic for flushing big buffers to storage from the commit 3fd8653b40 ... This should reduce memory usage when vmagent scrapes targets with big number of metrics and `-promscrape.streamParse` isn't enabled	2021-03-14 22:25:37 +02:00
Aliaksandr Valialkin	894246176f	lib/promscrape/discovery/kubernetes: do not start object watcher until initial objects are loaded	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	9e55db4a53	lib/promscrape: retry service discovery in a few seconds if it starts returning 0 targets ... This should reduce recovery time from temporary issues during service discovery	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	3b46ae1c05	lib/promscrape: remove duplicate target word in error message	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	b0b28eeb93	lib/promscrape/discovery/kubernetes: further optimize kubernetes service discovery for the case with many scrape jobs ... Do not re-calculate labels per each scrape job - reuse them instead for scrape jobs with identical Kubernetes role	2021-03-14 21:16:41 +02:00
Aliaksandr Valialkin	620f05cd2c	lib/promscrape/discovery: fixes after 133b288681 ... - Removed a deadlock in addAPIWatcher - Do not create unused ScrapeWork objects - Do not spend CPU resources on creating objectByKey map in addAPIWatcher This work is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1125	2021-03-13 15:22:38 +02:00
Aliaksandr Valialkin	54f902467d	lib/proxy: there is no need in cloning tlsCfg, which has been created two lines above	2021-03-12 10:48:01 +02:00
Aliaksandr Valialkin	72a8fa484b	lib/proxy: set proxy address in tls.Config.ServerName instead of the target address ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 10:41:25 +02:00
Aliaksandr Valialkin	60e0280a94	lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 03:36:11 +02:00
Aliaksandr Valialkin	b2732575f7	lib/storage: further tune filters sorting logic	2021-03-12 00:51:35 +02:00
Aliaksandr Valialkin	8fc29ffc67	lib/promscrape/discovery/kubernetes: use a single watcher per apiURL ... Previously multiple scrape jobs could create multiple watchers for the same apiURL. Now only a single watcher is used. This should reduce load on Kubernetes API server when many scrape job configs use Kubernetes service discovery.	2021-03-11 17:04:14 +02:00
Aliaksandr Valialkin	8b8d4cbcfe	lib/proxy: do not show inline basic auth passwords when logging errors related to proxy_url	2021-03-11 13:44:14 +02:00
Aliaksandr Valialkin	41f641b132	lib/promscrape/discovery/kubernetes: localize Bookmark parsing code ... This is a follow-up for e772d1c920	2021-03-11 13:08:56 +02:00
Aliaksandr Valialkin	6c9cd3f7c1	lib/promscrape/discovery/kubernetes: reduce load on Kubernetes API server by using watch bookmarks ... This allows continuing object watch from the last bookbark instead of reloading all the objects on watch errors or timeouts. See https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks	2021-03-10 15:08:40 +02:00
Aliaksandr Valialkin	bd8b7a88a7	lib/httpserver: export vm_available_memory_bytes and vm_available_cpu_cores metrics ... These metrics are useful for tracking the available memory and CPU cores for VictoriaMetrics apps.	2021-03-10 12:08:26 +02:00
Aliaksandr Valialkin	e15f3f4f2a	lib/proxy: pass proxy hostname in Host header of the CONNECT request ... This should resolve the following issue when connecting to tls proxy: cannot validate certificate for ... because it doesn't contain any IP SANs	2021-03-09 20:41:18 +02:00
Aliaksandr Valialkin	9d8223eafb	lib/proxy: set missing ServerName in TLS config for proxy_url. ... While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.	2021-03-09 19:01:14 +02:00
Nikolay	1310f84122	Changes tlsConfig init for proxy connections (#1121) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-09 19:01:13 +02:00
Aliaksandr Valialkin	0554430d7e	lib/promscrape: apply sample_limit after metric relabeling is applied as Prometheus does ... See the description for `sample_limit` option from Prometheus docs: Per-scrape limit on number of scraped samples that will be accepted. If more than this number of samples are present after metric relabeling the entire scrape will be treated as failed. 0 means no limit. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-03-09 15:52:41 +02:00
Aliaksandr Valialkin	7b66c8cbf8	lib/promscrape/discovery/kubernetes: remove too verbose logs about starting and stopping the watchers ... Log the number of objects loaded per each watch url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-09 15:07:12 +02:00
John Belmonte	edf39aa225	spelling fix: adjacent (#1115)	2021-03-09 09:19:16 +02:00
Aliaksandr Valialkin	502fab797a	lib/promscrape: add scrape_offset option to scrape_config ... This option can be used for specifying the particular offset per each scrape interval for target scraping	2021-03-08 11:59:32 +02:00
Aliaksandr Valialkin	c4a0bd5eac	lib/storage: go fmt	2021-03-08 11:59:31 +02:00
Aliaksandr Valialkin	c76a904bb0	lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow ... The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload	2021-03-07 21:17:48 +02:00
Aliaksandr Valialkin	c04505e585	lib/promscrape/discovery/kubernetes: reduce memory usage further when big number of scrape jobs are configured for the same kubernetes_sd_config role ... Serialize reloading per-role objects, so they don't occupy too much memory when objects for many scrape jobs are simultaneously refreshed. Do not reload per-role objects if they were already refreshed by concurrent goroutines. This should reduce load on Kubernetes API server when big number of scrape jobs are configured for the same Kubernetes role. This is a follow-up for 17b87725ed Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-07 20:03:22 +02:00
Aliaksandr Valialkin	175466bb41	lib/decimal: prevent exponent overflow when processing values close to zero ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1114	2021-03-05 18:53:41 +02:00
Aliaksandr Valialkin	5807ff57f3	lib/promscrape/discovery/kubernetes: reduce memory usage when Kubernetes service discovery is configured on a big number of scrape jobs ... Previously vmagent was creating a separate Kubernetes object cache per each scrape job. This could result in increased memory usage when monitoring a Kubernetes cluster with big number of objects (pods / nodes / services, etc.) as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113 Now it uses a shared map of scrape objects across multiple scrape jobs.	2021-03-05 17:32:33 +02:00
Aliaksandr Valialkin	92ddb8f197	lib/promscrape/discovery/kubernetes: move apiWatcher code to a separate file	2021-03-05 17:32:32 +02:00
Aliaksandr Valialkin	02c0959380	lib/promscrape: make cluster membership calculations consistent across 32-bit and 64-bit architectures	2021-03-05 09:06:08 +02:00
Aliaksandr Valialkin	133fb9fc00	lib/promscrape: add -promscrape.cluster.replicationFactor command-line flag for replicating scrape targets among vmagent instances in the cluster	2021-03-04 10:21:27 +02:00
Aliaksandr Valialkin	bae7a1b47a	lib/promscrape/discovery/kubernetes: fix tests after e154f4a644	2021-03-03 22:42:04 +02:00
Nikolay	7d92ef3acd	Fix ingress discovery api (#1110)	2021-03-03 10:45:50 +02:00
Aliaksandr Valialkin	25f453ce1a	lib/promscrape/discovery/kubernetes: properly check for nil pointer inside interface ... See https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1 This fixes a panic when the ScrapeWork is filtered out in swcFunc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1108	2021-03-03 10:42:54 +02:00
Aliaksandr Valialkin	fea27b9845	lib/promscrape: go fmt	2021-03-02 21:20:23 +02:00
Aliaksandr Valialkin	c67a07b469	lib/handshake: log read/write operation duration on connection errors ... This improve debuggability of network errors	2021-03-02 21:20:20 +02:00
Aliaksandr Valialkin	c8dde1fd6b	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:48:44 +02:00
Aliaksandr Valialkin	3fbe2bf1c8	lib/promscrape: pre-allocate space for a map in mergeLabels ... This should reduce the number of memory allocations when discovering big number of targets	2021-03-02 18:42:44 +02:00
Aliaksandr Valialkin	ac5c47a9f5	lib/promscrape/discovery: properly track vm_promscrape_discovery_kubernetes_objects_removed_total metric	2021-03-02 18:33:29 +02:00
Aliaksandr Valialkin	62d7e07ff7	lib/promrelabel: remove unneded optimizations for labeldrop and labelkeep actions ... These optimizations may slow down code execution by matching the same label against regexp two times instead of a single time	2021-03-02 18:01:08 +02:00
Aliaksandr Valialkin	f9c1fe3852	lib/promscrape/discovery/kubernetes: cache ScrapeWork objects as soon as the corresponding k8s objects are changed ... This should reduce CPU usage and memory usage when Kubernetes contains tens of thousands of objects	2021-03-02 16:44:19 +02:00
Aliaksandr Valialkin	f686174329	lib/promscrape/discovery/ec2: follow-up after f6114345de	2021-03-02 13:47:35 +02:00
Nikolay	fd8ca7df50	Adds webIndentity token for aws (#1099) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1080	2021-03-02 13:47:34 +02:00
Aliaksandr Valialkin	e45c399467	lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format ... Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.	2021-03-02 13:22:10 +02:00
Aliaksandr Valialkin	f4969a624d	lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as foo; 1 2 ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100	2021-03-01 17:17:01 +02:00
Aliaksandr Valialkin	b89a4fac2f	lib/promscrape/discovery/kubernetes: deflake tests; a follow-up for 05fb08713c	2021-03-01 14:31:44 +02:00
Aliaksandr Valialkin	c3bf72992f	lib/promscrape: explicitly stop and cleanup service discovery routines when new config is read from -promscrape.config ... This should reduce memory usage when `-promscrape.config` file frequently changes	2021-03-01 14:15:16 +02:00
Aliaksandr Valialkin	8af9370bf2	lib/promscrape: use target arg in ScrapeWork cache	2021-03-01 12:31:11 +02:00
Aliaksandr Valialkin	5328506c3e	lib/promscrape: typo fix, which prevented from caching ScrapeWork entries	2021-03-01 12:13:13 +02:00
Aliaksandr Valialkin	d5058caccc	lib/promscrape: add vm_promscrape_scrapework_cache_* metrics for tracking ScrapeWork cache effectiveness	2021-03-01 12:05:58 +02:00
Aliaksandr Valialkin	ee5d26a546	lib/httpserver: make make errcheck happy after the commit 9fc7726d84	2021-03-01 00:35:30 +02:00
Aliaksandr Valialkin	109bfaadad	lib/promscrape: reduce CPU usage an memory allocations when constructing scrapeWorkKey	2021-02-28 22:30:36 +02:00
Aliaksandr Valialkin	9e644ef111	lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls ... This should fix the `http: superfluous response.WriteHeader call` issue See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078	2021-02-28 19:23:26 +02:00
Aliaksandr Valialkin	9a2bf65134	lib/promscrape: add ability to spread scrape targets among multiple vmagent instances ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084	2021-02-28 18:40:42 +02:00
Aliaksandr Valialkin	3e44d9947e	lib/promscrape/discovery/kubernetes: properly account the number of objects when watcher is stopped ... A follow-up for b21b110b7a	2021-02-28 17:06:49 +02:00
Aliaksandr Valialkin	0ef7a94056	lib/promscrape/discovery/kubernetes: add vm_promscrape_discovery_kubernetes_* metrics for monitoring internal state of k8s service discovery	2021-02-28 16:58:45 +02:00
Aliaksandr Valialkin	f52bdbe2a3	lib/promscrape/discovery/kubernetes: remove resourceVersionMatch=NotOlderThan query arg when watching for k8s object changes, since it cannot be used when watch=1 query arg is passed	2021-02-28 16:08:43 +02:00
Aliaksandr Valialkin	6f8866a40a	lib/promscrape: fix possible deadlock in parallel execution of target relabeling	2021-02-28 16:05:32 +02:00
Aliaksandr Valialkin	5c9e657808	lib/promscrape/discovery/kubernetes: fix deadlock in startWatcherForURL ... reloadObjects must be called without holding aw.mu lock	2021-02-28 15:25:33 +02:00
Aliaksandr Valialkin	e77f2f8630	lib/promscrape/discovery/kubernetes: typo fix after 241ffd1f3b	2021-02-28 15:15:27 +02:00
Aliaksandr Valialkin	82441537ff	lib/promscrape/discovery/kubernetes: pre-populate labelsByKey in reloadObject()	2021-02-28 15:09:43 +02:00
Aliaksandr Valialkin	e003453941	lib/promscrape/discovery/kubernetes: compare sorted sets of labels in tests ... This should deflake tests where the order of labels isn't stable	2021-02-28 14:12:32 +02:00
Aliaksandr Valialkin	6a21ef87b7	lib/promscrape: add missing startWatchersForRole() call at the beginning of apiWatcher.getLabelsForRole	2021-02-28 14:00:00 +02:00
Aliaksandr Valialkin	6d0e7fb8b0	lib/promscrape/discovery/kubernetes: reload k8s resources on every error ... This is needed for obtaining fresh resourceVersion	2021-02-27 01:46:59 +02:00
Aliaksandr Valialkin	7f1302688f	lib/fs: follow-up after f3a03c4164	2021-02-27 01:09:37 +02:00
Nikolay	d88fa5ebe4	Adds windows build (#1040) ... * fixes windows compilation, adds signal impl for windows, adds free space usage for windows, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036 NOTE victoria metrics database still CANNOT work under windows system, only vmagent is supported. To completly port victoria metrics, you have to fix issues with separators, parsing and posix file removall * rollback separator * Adds windows setInformation api, it must behave like unix, need to test it. changes procutil * check for invlaid param * Fixes posix delete semantic * refactored a bit * fixes openbsd build * removed windows api call * Fixes code after windows add * Update lib/procutil/signal_windows.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-27 01:06:22 +02:00
Aliaksandr Valialkin	44975e28fe	lib/promscrape: yet another typo fix after ed8441ec52	2021-02-26 23:36:34 +02:00
Aliaksandr Valialkin	c1b8729bd8	lib/fs: properly handle stale NFS file handle error during file deletion ... This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.	2021-02-26 23:24:46 +02:00
Aliaksandr Valialkin	50e74c439c	lib/promscrape: typo fix after ed8441ec52	2021-02-26 23:04:40 +02:00
Aliaksandr Valialkin	fa3ce450fb	lib/promscrape: cache ScrapeWork ... This should reduce the time needed for updating big number of scrape targets.	2021-02-26 21:43:41 +02:00
Aliaksandr Valialkin	efcdf613c2	lib/promscrape/discovery/kubernetes: cache target labels ... This should reduce CPU usage on repeated SDConfig.GetLabels() calls.	2021-02-26 20:24:29 +02:00
Aliaksandr Valialkin	22822feea3	lib/promscrape/discovery/kubernetes: errcheck fix	2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin	dc8c045378	lib/promscrape: cleanup after 9b2246c29b ... Main points: * Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit * Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001 * Corner case fixes.	2021-02-26 19:09:12 +02:00
Nikolay	cf9262b01f	vmagent kubernetes watch stream discovery. (#1082) ... * started work on sd for k8s * continue work on watch sd * fixes * continue work * continue work on sd k8s * disable gzip * fixes typos * log errror * minor fix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin	4cfac70cde	lib/promscrape: remove duplicate code a bit	2021-02-26 15:54:07 +02:00
Aliaksandr Valialkin	55eb983193	lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel	2021-02-26 12:48:48 +02:00
Aliaksandr Valialkin	f8baf1a76d	lib/promrelabel: optimize labeldrop and labelkeep relabeling for prefix.* and prefix.+ regexps	2021-02-24 17:58:11 +02:00
Aliaksandr Valialkin	06676c8feb	lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache	2021-02-23 15:50:08 +02:00
faceair	b1409a7413	lib/storage: correct tagfilter match cost (#1079)	2021-02-22 21:54:37 +02:00
Aliaksandr Valialkin	197ecca426	lib/promrelabel: add more optimizations for relabeling for common cases	2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin	63c16c3fdf	lib/promrelabel: optimize relabeling performance for common cases	2021-02-22 00:51:07 +02:00
Aliaksandr Valialkin	8b87398333	lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric	2021-02-21 23:21:35 +02:00
Aliaksandr Valialkin	587132555f	lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation ... This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now, so GC doesn't need visiting them.	2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin	bd3bcdc43c	lib/storage: do not re-calculate stats for heavy tag filters ... This should reduce the number of slow queries when stats for heavy tag filters was recalculated.	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	b8a5ee2e93	lib/{mergeset,storage}: allow merging smaller number of small parts ... While this may increase CPU and disk IO usage needed for background merge, this also recudes CPU usage during queries in production. This is because such queries tend to read recently added data and it is better to have lower number of parts for such data in order to reduce CPU usage. This partially reverts ebf8da3730	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	34195218e1	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	f09b46b2b1	lib/promscrape: typo fix after the commit f26162ec99	2021-02-19 00:34:18 +02:00
Aliaksandr Valialkin	72eef964d9	app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065	2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin	502d0e2524	lib/promscrape: add scrape_align_interval config option into scrape config ... This option allows aligning scrapes to a particular intervals.	2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin	10ccb92e4d	lib/storage: use composite index for a query with a name filter and negative filters	2021-02-18 18:57:45 +02:00
Aliaksandr Valialkin	418de71509	lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters ... Example: `node_cpu_seconds_total{mode!="idle"}`	2021-02-18 18:33:05 +02:00
Aliaksandr Valialkin	628b8eb55e	lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. ... This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters	2021-02-18 14:01:18 +02:00
Aliaksandr Valialkin	fd41f070db	lib/storage: sort tag filters by the number of loops they need for the execution ... This metric should work better than the filter execution duration, since it cannot be distorted by concurrently running queries.	2021-02-18 12:52:29 +02:00
Aliaksandr Valialkin	9566015a36	Revert "lib/mergeset: tune lifetime for entries inside block caches" ... This reverts commit 458c89324d. Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.	2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin	b4c7d5992b	lib/storage: move composite filters to the top during sorting	2021-02-17 20:26:32 +02:00
Aliaksandr Valialkin	2005d8212a	lib/storage: return back filter arg to getMetricIDsForTagFilter function ... The filter arg has been removed in the commit c7ee2fabb8 because it was preventing from caching the number of matching time series per each tf. Now the cache contains duration for tf execution, so the filter shouldn't break such caching.	2021-02-17 19:33:15 +02:00
Aliaksandr Valialkin	83da939947	app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics	2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin	0c5bb2a397	lib/storage: revert ecf132933e, since negative filters require the same amount of work as non-negative filters	2021-02-17 18:56:13 +02:00
Aliaksandr Valialkin	bbc287ea6a	lib/storage: tag filters sorting...	2021-02-17 18:56:12 +02:00
Aliaksandr Valialkin	f5e841c1e9	lib/storage: further tune tag filters sorting	2021-02-17 17:28:36 +02:00
Aliaksandr Valialkin	9d1f14d94c	lib/storage: tune the logic for sorting tag filters according the their exeuction times	2021-02-17 15:02:19 +02:00
Aliaksandr Valialkin	1a19702d92	lib/storage: make sure that nobody uses partitions when closing the table	2021-02-17 15:02:18 +02:00
Aliaksandr Valialkin	3062ff0fdb	app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration ... The metrics are: - vm_vmselect_http_requests_total{accountID="...",projectID="..."} - the total number of select requests per each tenant - vm_vmselect_http_duration_ms_total{accountID="...",projectID="..."} - the total duration in milliseconds for per-tenant select requests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932	2021-02-16 23:30:29 +02:00
Aliaksandr Valialkin	3b8596bfec	lib/httpserver: make errcheck happy	2021-02-16 22:05:59 +02:00
Aliaksandr Valialkin	1ab7a8dfd5	lib/storage: more tuning for tag filters sorting according the time they take	2021-02-16 21:27:27 +02:00
Aliaksandr Valialkin	35a23234ca	lib/mergeset: tune lifetime for entries inside block caches ... This should reduce memory usage in general case without significant CPU usage increase	2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin	500acb958c	lib/mergeset: clarify comments in the code a bit	2021-02-16 18:03:11 +02:00
Aliaksandr Valialkin	da05c10544	lib/uint64set: remove memory allocation in bucket16.appendTo when sorting smallPool	2021-02-16 15:31:59 +02:00
Aliaksandr Valialkin	18305caadb	lib/httpserver: cache /metrics output for a second ... This should reduce CPU load when `/metrics` output is scraped with a frequency exceeding a request per second	2021-02-16 14:57:13 +02:00
Aliaksandr Valialkin	7869d38043	lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names	2021-02-16 13:59:11 +02:00
Aliaksandr Valialkin	d788876a7a	lib/mergeset: remove unused code after a4140de9e6	2021-02-16 13:40:29 +02:00
Aliaksandr Valialkin	55952f8f2e	lib/storage: tune sorting for tag filters	2021-02-16 13:07:42 +02:00
Aliaksandr Valialkin	3eae03a337	lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs	2021-02-15 16:39:52 +02:00
Aliaksandr Valialkin	46e98ed490	vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 ... The new version switches from log-linear histograms to log-based histograms, which provide up to 3.6 times better accuracy.	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	93ff866e91	lib/storage: reduce the minimum supported retention for inverted index from one month to one day	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	5d5d310dcc	lib/flagutil: prevent from integer overflow when parsing duration	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	fccb481de2	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_* and __meta_kuberntes_endpoints_annotation_* labels to role: endpoints ... This syncs kubernetes SD with Prometheus 2.25 See 617c56f55a	2021-02-15 02:51:36 +02:00
Aliaksandr Valialkin	54a09de037	lib/logger: explicitly import "time/tzdata" package for embedding tzdata into the app ... The approach with `timetzdata` build tag didn't work for GOARCH=arm and GOARCH=ppc64le due to the issue https://github.com/golang/go/issues/44073#issuecomment-778854298	2021-02-15 01:00:30 +02:00
Aliaksandr Valialkin	6f3bbf21b8	lib/storage: sort tag filters by actual execution time instead of by the number of matching time series ... This should improve query speed for queries with regexp filters matching small number of time series on a label with big number of unique values.	2021-02-15 00:19:46 +02:00
Aliaksandr Valialkin	9e3993c585	lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. ... For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` . This could result in missing time series for such tag filters.	2021-02-14 23:39:19 +02:00
Aliaksandr Valialkin	18c2075159	lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics ... These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes. * It is better using `up == 0` query for determining failing targets. * It is better using the following query for determining targets with exceeded limit on the number of metrics: scrape_samples_scraped > 0 if up == 0	2021-02-12 05:23:27 +02:00
Aliaksandr Valialkin	4e645a5fd3	lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case	2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin	eeb92eb7fc	lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata	2021-02-10 17:55:51 +02:00
Aliaksandr Valialkin	08f21d8761	app/vmstorage: export vm_composite_index_min_timestamp metric	2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin	b27288f1b0	lib/storage: parallelize tag filters execution a bit ... This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.	2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin	4262c2f7c2	lib/storage: remove filter arg from getMetricIDsForDateTagFilter function ... The `filter` arg breaks the logic for sorting tag filters by the matching metrics, which may result in non-optimal performance during time series search.	2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin	681dfb7485	lib/storage: fix inconsistencies in error logs	2021-02-10 16:32:21 +02:00
Aliaksandr Valialkin	148422bcba	lib/storage: disable composite index usage when querying old data	2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin	17d5a03f6e	lib/storage: fix metric name match for composite filter	2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin	fa0ef143b1	lib/storage: optimize search by label filters matching big number of time series	2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin	5c9715a89a	lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day ... This should help systems with multiple CPU cores	2021-02-10 00:04:19 +02:00
Aliaksandr Valialkin	082eabf51e	lib/fs: remove the code for tracking whether the given memory region is in page cache ... This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.	2021-02-09 16:51:11 +02:00
Aliaksandr Valialkin	afa9cf9c57	lib/mergeset: remove dead code left after a4140de9e6	2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin	9ed7789fef	optimize Storage.updatePerDateData()	2021-02-09 02:59:53 +02:00
Aliaksandr Valialkin	ea328b7391	lib/storage: skip deduplication when creating inmemory data blocks ... The deduplication will be performed later during merging such blocks.	2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin	7b7963a77f	lib/mergeset: unconditionally cache indexdb blocks ... Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage. This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.	2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin	e8ee9fa7fe	app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches	2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin	2b36eb3d82	lib/cgroup: follow-up after b9bf3cbe3e	2021-02-08 16:01:26 +02:00
Nikolay	32603f57cc	refactored cgroups limits, (#1061) ... adds tests, remove os.Exec	2021-02-08 16:01:26 +02:00
Aliaksandr Valialkin	2dbb12563b	lib/storage: optimize data ingestion in the beginning of every hour ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-08 12:04:51 +02:00
Aliaksandr Valialkin	1a6eb0c3cf	lib/logger: exit the app if unsupported timezone value has been passed to -loggerTimezone ... While at it, clarify descrption for `-loggerTimezone` command-line flag.	2021-02-07 23:33:35 +02:00
Aliaksandr Valialkin	c6a7288109	lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion ... This should reduce possible CPU usage spikes at the beginning of every hour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-04 18:46:23 +02:00
Aliaksandr Valialkin	b5eba70595	lib/httpserver: expose process_open_fds and process_max_fds metrics ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037	2021-02-04 16:42:32 +02:00
Nikolay	6fc2a8e544	fixes dockerswarm (#1053) ... fixes improper usage of host network services https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028	2021-02-04 15:57:41 +02:00
Aliaksandr Valialkin	8249f13104	app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards ... Example: foo{bar{x,yz},a[b-c],*de}	2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin	2976ec89b8	lib/storage: fix a bug, which breaks searching by Graphite wildcard filters	2021-02-03 20:15:50 +02:00
Aliaksandr Valialkin	45a63a1da9	sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks	2021-02-03 20:15:37 +02:00
Aliaksandr Valialkin	4b930b9ffe	app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax	2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin	755b0998ce	lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric	2021-02-02 16:16:51 +02:00
Aliaksandr Valialkin	4c59dbc127	lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics	2021-02-01 20:06:16 +02:00
Aliaksandr Valialkin	e79799a784	lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func	2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin	fdf9de98f8	app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values ... This commit also adds --vm-round-digits command-line option to vmctl tool.	2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin	2c14c70903	lib/logger: initialize timezone by UTC in order to fix failing tests	2021-01-27 01:00:57 +02:00
Aliaksandr Valialkin	ffec5131ae	lib/promscrape: export vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics ... These metrics could be useful for determining imporperly working scrape targets. Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	4b324da947	all: consistently use timers from timerpool	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	588090765c	lib/fs: properly initialize cleaner for pageCache bitmaps ... Previously it wasnt working because the timer was fired only once	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	68a66be811	lib/logger: add -loggerTimezone command-line flag for adjusting timezone for timestamps in log messages	2021-01-26 22:53:25 +02:00
Aliaksandr Valialkin	fdced59278	lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection	2021-01-22 13:22:59 +02:00
faceair	cf76d8dd79	lib/mergeset: add missing shouldCacheBlock (#1019)	2021-01-15 11:47:25 +02:00
Aliaksandr Valialkin	81789da731	lib/backup: increase backup chunk size from 128MB to 1GB ... This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing	2021-01-13 12:16:39 +02:00
Nikolay	095be83f2f	bumps minimal tls version (#1012)	2021-01-13 00:39:09 +02:00
Aliaksandr Valialkin	3d79471fb3	lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here ... This is a follow-up commit after c8ea697db8	2021-01-12 15:20:22 +02:00
Aliaksandr Valialkin	719ad49adf	lib/storage: de-duplicate tags in MetricName.sortTags ... Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.	2021-01-12 15:03:22 +02:00
Nikolay	9f0a4fd00e	Fixes error handling for promscrape.streamParse (#1009) ... properly return error if client cannot read data, properly suppress scraper errors	2021-01-12 13:35:09 +02:00
Aliaksandr Valialkin	c97681b45c	lib/promscrape: properly show scrape duration on /targets page ... Previously it has been shown as 0.000s for any scrape duration.	2021-01-11 21:15:50 +02:00
Aliaksandr Valialkin	4ee53c3961	all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000	2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin	d5a2b120e9	app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage	2021-01-08 00:12:12 +02:00
Aliaksandr Valialkin	ca8919e8e1	lib/storage: wait for pending transactions before closing and dropping the partition ... This deflakes `make test-full-386` test	2020-12-25 11:46:47 +02:00
Aliaksandr Valialkin	c0511144e3	lib/storage: physically remove stale parts ... Previously they were removed from partition struct, but the corresponding directories weren't removed. This is a follow-up for 46dba00756	2020-12-24 16:56:09 +02:00
Nikolay	dd2f815204	fixes kubernetes_sd (#983) ... * fixes kubernetes_sd, adds missing service metadata for pod ports without endpoint https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982 * fix test	2020-12-24 11:34:12 +02:00
Aliaksandr Valialkin	367fc17933	lib/promscrape: code prettifying for 8dd03ecf19	2020-12-24 10:57:20 +02:00
Nikolay	b00f7816e2	adds proxy_url support, (#980) ... * adds proxy_url support, adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery, adds proxy_url to the scrape_config for targets scrapping, http based proxy is supported atm, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503 * fixes imports	2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin	66f8fbbb32	lib/storage: do not remove parts outside the configured retention if they are currently merged ... These parts are automatically removed after the merge is complete.	2020-12-24 09:02:12 +02:00
Aliaksandr Valialkin	fa3bcf220f	lib/storage: remove stale parts as soon as they go outside the configured retention ... Previously such parts could remain undeleted for long durations until they are merged with other parts. This should help for `-retentionPeriod` values smaller than one month.	2020-12-22 19:55:07 +02:00
Aliaksandr Valialkin	6859737329	lib/storage: properly determine max rows for output part when merging small parts	2020-12-18 23:26:28 +02:00
Aliaksandr Valialkin	edbe35509e	lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage	2020-12-18 20:01:20 +02:00
Aliaksandr Valialkin	1ee5a234dc	lib/promscrape: remove ID field from ScrapeWork struct. Use a pointer to ScrapeWork as a key in targetStatusMap ... This simplifies the code a bit.	2020-12-17 14:31:55 +02:00
Aliaksandr Valialkin	4fd2973e7c	lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2 ... Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.	2020-12-16 23:42:17 +02:00
BigFish	60dd48c9eb	lib/protoparser/prometheus/parser.go (#970) ... fix parse timestamp error if there are some whitespaces after timestamp	2020-12-16 23:42:16 +02:00
Aliaksandr Valialkin	c4ec977934	lib/promscrape: properly remove deleted target from /targets page ... Previously `sw` variable wasn't captured correctly by the started goroutine.	2020-12-15 20:58:37 +02:00
Aliaksandr Valialkin	eddc2bd017	lib/promscrape: properly handle scrape errors when stream parsing is enabled ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967	2020-12-15 14:10:52 +02:00
Nikolay	7064c4eb8e	adds new Array Flags (#965) ... * adds ArrayDuration and ArrayBool flags, makes sendTimeout and tlsInsecure configurable per remoteWrite url * added backward compatibility testcases for ArrayDuration and ArrayBool * fixes bool flag * fixes test cases	2020-12-15 12:59:33 +02:00
Aliaksandr Valialkin	104aac170e	lib/promscrape: add bootstrap styles to /targets html page	2020-12-15 12:38:29 +02:00
Aliaksandr Valialkin	ad961fe1f1	lib/promscrape: formatting fixes for /tarets page	2020-12-15 11:59:28 +02:00
Aliaksandr Valialkin	38145cfbb8	lib/promscrape: formatting fixes for /targets page	2020-12-15 11:27:22 +02:00
Aliaksandr Valialkin	d98a2f217b	lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964	2020-12-14 19:25:53 +02:00
Aliaksandr Valialkin	b904ae3722	lib/promscrape: add missing whitespace between duration and ago word at /targets page	2020-12-14 14:20:30 +02:00
Aliaksandr Valialkin	a2eb451de4	app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser	2020-12-14 14:13:01 +02:00
Nikolay	324e3aa1a5	Changes targets api (#961) ... * changes /targets api adds html response if requester accepts text/html, adds quick template for /targets api, fixes pathPrefix for / requests * changes namings * renamed targets file * Update app/victoria-metrics/main.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds trimspace to qtpl, moves content-type for targets response closer to writer * fixes bug with prefix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-14 14:13:00 +02:00
Aliaksandr Valialkin	c80d38f00c	lib/promscrape/discovery/consul: reduce load on Consul API server by increasing timeout for blocking requests from 50 seconds to 9 minutes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-11 17:26:34 +02:00
Aliaksandr Valialkin	a84467958a	lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server ... Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg. See https://www.consul.io/api-docs/health#list-nodes-for-service for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170	2020-12-08 21:53:23 +02:00
Aliaksandr Valialkin	1a237c6903	all: properly handle CPU limits set on the host system/container ... This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin	38188e1d6b	lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork() ... This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they are no longer used. See the related commit e205975716 and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-12-08 17:55:21 +02:00
Aliaksandr Valialkin	d5faad0240	lib/promscrape: re-use strings for labels stored in ScrapeWork ... This should reduce memory usage when working with big number of scrape targets.	2020-12-08 12:23:44 +02:00
Aliaksandr Valialkin	06091cfdf8	lib/promscrape: export vm_promscrape_scrapers_{started|stopped}_total metrics for monitoring target churn rate	2020-12-08 11:58:44 +02:00
Aliaksandr Valialkin	affcee199c	lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value ... This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.	2020-12-08 11:52:20 +02:00
Aliaksandr Valialkin	56a0b058c1	lib/promscrape: export vm_promscrape_active_scrapers{type="<sd_type>"} metric for tracking the number of active scrapers per each service discovery type	2020-12-08 01:54:44 +02:00
Aliaksandr Valialkin	b5b32c65b0	lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun command-line flag is passed ... Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944	2020-12-07 13:18:16 +02:00
Aliaksandr Valialkin	9d79d11a6c	lib/promscrape: mention in scrape error message that scrape errors can be disabled by -promscrape.suppressScrapeErrors command-line flag	2020-12-06 23:27:07 +02:00
Aliaksandr Valialkin	f6d32f99d7	lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set	2020-12-06 13:19:32 +02:00
Aliaksandr Valialkin	3d00613076	lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol	2020-12-06 12:00:28 +02:00
Aliaksandr Valialkin	1430bbcf33	lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries ... Too low limit could result in unexpected errors when performing big number of blocking queries.	2020-12-05 12:15:47 +02:00
Aliaksandr Valialkin	528587deef	lib/flagutil: make golangci-lint happy by using strings.TrimPrefix instead of manual prefix removal via strings.HasPrefix	2020-12-03 22:07:26 +02:00
Aliaksandr Valialkin	bdac2171f1	all: do not print usage info for all the flags when incorrect command-line flag is passed ... This should improve usability for VictoriaMetrics apps that have big number of command-line flags, i.e. all the apps.	2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin	96190f9d45	lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher	2020-12-03 20:14:48 +02:00
Aliaksandr Valialkin	4e4a93c586	lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header	2020-12-03 20:05:54 +02:00
Aliaksandr Valialkin	7a889f6850	lib/promscrape: code cleanup after c6dee6c52d ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-03 19:52:09 +02:00
Nikolay	0b302d33cb	Changes consul discovery api (#921) ... * adds consul watch api, it must reduce load on consul service with blocking wait requests, changed discoveryClient api with fetchResponseMeta callback. * small fix * fix after master merge * adds watch client at discovery utils * fixes consul watcher, changes namings, fixes data race * small typo fix * sanity fix * fix naming and service node update	2020-12-03 19:52:08 +02:00
Aliaksandr Valialkin	9eca96596f	lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test	2020-11-29 01:25:50 +02:00
Aliaksandr Valialkin	2385ac11c0	lib/promscrape: fix failing tests after a906b3862f	2020-11-29 01:25:49 +02:00
Aliaksandr Valialkin	2ed721e457	lib/protoparser/prometheus: properly parse OpenMetrics timestamps ... OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds. This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.	2020-11-27 14:54:36 +02:00
Aliaksandr Valialkin	3bb9bf33d6	lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets	2020-11-26 18:32:16 +02:00
Aliaksandr Valialkin	af667c59c1	all: typo fix: thouthand->thousand	2020-11-26 13:34:05 +02:00
Aliaksandr Valialkin	3dd2282ed9	lib/promscrape: release http response non-200 status code	2020-11-26 13:25:25 +02:00
Aliaksandr Valialkin	3f52e59efe	app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun	2020-11-25 23:01:39 +02:00
Aliaksandr Valialkin	8f8727cb65	lib/mergeset: tune the number of rawItemsBlocks to merge at once ... 512 blocks give higher ingestion performance and slightly lower memory usage	2020-11-25 21:53:15 +02:00
Aliaksandr Valialkin	8fcd87a6a5	lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset	2020-11-25 21:20:02 +02:00
Aliaksandr Valialkin	03002f1fe1	lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention ... This should simplify debugging when the source of the metric with unexpected timestamp must be found.	2020-11-25 14:44:29 +02:00
Aliaksandr Valialkin	4848a05924	lib/storage: typo fix in error message: allowd->allowed	2020-11-25 14:15:54 +02:00
Aliaksandr Valialkin	26e699c440	lib/protoparser/prometheus: properly parse "infinity" values in OpenMetrics format ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924	2020-11-24 19:02:50 +02:00
Aliaksandr Valialkin	a8b5a6e517	lib/logger: disable rate limiting for error and warn logs by default	2020-11-24 12:42:07 +02:00
Aliaksandr Valialkin	7f3e884a31	all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9	2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin	dad8b76a0e	lib/protoparser/prometheus: properly parse metrics with exemplars ... Examplars have been introduced in OpenMetrics - see https://github.com/OpenObservability/OpenMetrics/blob/master/OpenMetrics.md#exemplars-1 Previously VictoriaMetrics couldn't parse the following metric foo{bar="baz"} 123 # exemplar here This commit fixes this. Note that VictoriaMetrics ignores the exemplar as for now.	2020-11-24 12:30:34 +02:00
Aliaksandr Valialkin	8b82f9d8b8	lib/promscrape: expose __meta_ec2_ipv6_addresses label for ec2_sd_config like Prometheus will do in the next release	2020-11-23 16:57:03 +02:00
Aliaksandr Valialkin	c2186279b7	lib/promscrape: add filters option to dockerswarm_sd_config like Prometheus did in v2.23.0	2020-11-23 16:27:33 +02:00
Aliaksandr Valialkin	f4fd917e4f	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin	d892d63204	lib/promscrape: hint that -enableTCP6 command-line flag can be used for connecting to IPv6 addresses	2020-11-21 14:39:05 +02:00
Aliaksandr Valialkin	8608e956dd	lib/promscrape/discovery/eureka: follow-up after eec76718e9	2020-11-20 14:02:14 +02:00
Nikolay	bb2bcb9725	Adds eureka service discovery (#913) ... * Adds eureka service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851 Netflix service discovery for AWS * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-20 14:02:13 +02:00
Aliaksandr Valialkin	8e1f657ef9	lib/logger: follow-up for 09105ff49c	2020-11-19 12:37:05 +02:00
Nikolay	f54a5f3868	Adds log suppression per caller (#908) ... * Adds log suppression per caller https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905 * fixes style and report message	2020-11-19 12:19:20 +02:00
Aliaksandr Valialkin	0895b7f411	lib/logger: add -loggerWarnsPerSecondLimit command-line flag for rate limiting of WARN log messages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905	2020-11-18 03:43:17 +02:00
Aliaksandr Valialkin	7d76fdedcc	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place ... This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin	a9287cf564	lib/storage: do not pass (accountID, projectID) to SearchTagNames(), since they are already passed via tfss	2020-11-16 18:04:30 +02:00
Aliaksandr Valialkin	ac7460abdd	lib/storage: add a test for Storage.SearchMetricNames	2020-11-16 13:18:48 +02:00
Aliaksandr Valialkin	eea1be0d5c	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:52:23 +02:00
Aliaksandr Valialkin	4be5b5733a	app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb	2020-11-16 02:40:04 +02:00
Aliaksandr Valialkin	9ec964bff8	lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page	2020-11-16 00:44:54 +02:00
Aliaksandr Valialkin	f80d6473e1	lib/protoparser/promremotewrite: log the time spent on unsuccessful data read from the network ... This should help with debugging `connection timed out` errors.	2020-11-13 17:49:21 +02:00
Vasily	8ba168f3be	Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig (#796) ... * Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig * Add omitempty annotation to all the default/optional values * Fix annotations after review	2020-11-13 16:17:03 +02:00
Aliaksandr Valialkin	d21b1606a1	lib/protoparser/opentsdbhttp: increment errors counter on unmarshal errors ... This is a follow-up for 149c0c4a6d	2020-11-13 13:23:11 +02:00
Aliaksandr Valialkin	22c1e29284	lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols ... The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 13:05:34 +02:00
Aliaksandr Valialkin	9dfe00c962	lib/protoparser/promremotewrite: synchronously process Prometheus remote_write requests ... There is no reason in processing these requests asynchronously in the face of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896 Synchronous processing code is easier to read and understand than the previous async code	2020-11-13 12:17:32 +02:00
Aliaksandr Valialkin	739b88c1e4	lib/protoparser/promremotewrite: forward errors, which can occur during data ingestion, to the caller of ParseStream, so it could properly return HTTP 503 status code on non-nil error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 11:00:41 +02:00
Aliaksandr Valialkin	7ceaf4ba8f	all: consistently return text-based HTTP responses with charset=utf-8 ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:30:21 +02:00
immerrr again	1ec1a9f27f	app/vmstorage: add "/internal/force_flush" endpoint (#893)	2020-11-11 14:46:37 +02:00
Aliaksandr Valialkin	697fd44158	lib/promscrape: make a copy of ScrapeWork from discovered []ScrapeWork slice instead of referring to an item in this slice ... This should prevent from holding previously discovered []ScrapeWork slices when a part of discovered targets changes over time. This should reduce memory usage for the case when big number of discovered scrape targets changes over time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 16:13:31 +02:00
Aliaksandr Valialkin	2ec02b7bdb	lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets ... This should reduce load on GC a bit when discovering big number of scrape targets	2020-11-10 15:57:43 +02:00
Aliaksandr Valialkin	a8562d643b	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin	aa3e46a02d	lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-09 10:55:03 +02:00
Aliaksandr Valialkin	b8083b7659	lib/promscrape: clean references to label name and label value strings after applying per-target relabeling ... This should reduce memory usage when per-target relabeling creates big number of temporary labels with long names and/or values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin	b4efe626d7	lib/promscrape/discovery/kubernetes: go fmt	2020-11-07 13:04:09 +02:00
Aliaksandr Valialkin	92bc1afcee	lib/promscrape/discovery/kubernetes: reduce memory usage for labels when discovering big number of scrape targets by using string concatenation instead of fmt.Sprintf ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 13:03:01 +02:00
Aliaksandr Valialkin	535fea3d11	lib/promscrape: eliminate data race in stream parse mode ... Previously `-promscrape.streamParse` mode could result in garbage labels for the scraped metrics because of data race. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247	2020-11-07 12:45:52 +02:00
Aliaksandr Valialkin	72011bcc45	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin	f2bff64933	lib/storage: remove data race when updating rowsDeleted	2020-11-05 01:19:30 +02:00
Aliaksandr Valialkin	c5e6c5f5a6	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:19:29 +02:00
Nikolay	5b235b902b	Adds ready probe (#874) ... * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:33:48 +02:00
Aliaksandr Valialkin	2cd86d0220	lib/promscrape: docs update after e4182dd896 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 17:13:34 +02:00
Nikolay	d0a9b24c5a	reduces memory usage for vmagent, (#880) ... * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin	c736339843	lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively ... Previously such blocks were cleaned after they weren't accessed during 10 minutes. Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.	2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin	f35aafb6a5	Revert "lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets" ... This reverts commit b08c6f5144.	2020-11-04 11:45:38 +02:00
Aliaksandr Valialkin	b08c6f5144	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 11:09:05 +02:00
Aliaksandr Valialkin	c046735571	lib/promscrape: reduce memory allocations in promLabelsString() function ... This should help with reducing memory usage in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 10:38:59 +02:00
Aliaksandr Valialkin	c0bd208c77	lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification	2020-11-03 15:32:09 +02:00
Aliaksandr Valialkin	1b9778a756	lib/storage: remove unneeded fmt.Sprintf	2020-11-03 14:21:04 +02:00
John Belmonte	8653e2658e	add short_version label to vm_app_version metric (#877) ... * add short_version label to vm_app_version metric use case: Version panel of Grafana dashboard should use a live query, but currently it uses a template query which becomes stale. Grafana is not able to preform regex substitution on labels. * Update metrics.go * fix compile	2020-11-03 14:12:42 +02:00
Aliaksandr Valialkin	f3a7e6f6e3	lib/storage: remove obsolete code	2020-11-02 19:17:30 +02:00
Aliaksandr Valialkin	3ed9f1d5a9	lib/promscrape: properly handle response body after 301 redirect ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:59 +02:00
Nikolay	e8fe618bbb	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-11-01 23:25:23 +02:00
Nikolay	058f49de57	adds leading forward slash check for scrapeURL path (#855) ... * fixes in-consistency with prometheus behaviour for scrape targets url path. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835	2020-11-01 23:22:16 +02:00
Aliaksandr Valialkin	ed724d25ba	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin	901514be88	lib/storage: drop more samples outside the given retention during background merge ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-31 20:44:47 +02:00
Aliaksandr Valialkin	abdf22e0bb	app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets ... This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin	7599e5c835	lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*	2020-10-20 20:09:52 +03:00
Aliaksandr Valialkin	9c5cd5a6c5	lib/storage: code cleanup after 5bfd4e6218	2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin	0db7c2b500	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:42:46 +03:00
Aliaksandr Valialkin	5518b11720	lib/memory: do not print trailing zeroes in logs for -memory.allowedPercent command-line flag	2020-10-20 14:42:37 +03:00
faceair	0093ee3cd9	disable response compression on websocket (#841)	2020-10-17 13:33:37 +03:00
Aliaksandr Valialkin	efb1989193	lib/storage: small code adjustements after d2960a20e0 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-10-17 01:17:12 +03:00
faceair	8ddf089deb	evaluate the execution cost of all tag filters (#824) ... * evaluate the execution cost of all tag filters * fix suffixes typo	2020-10-17 01:13:20 +03:00
Nikolay Khramchikhin	a5d842caf8	fixes openstack api endpoint with suffix trim adds openstack (#840) ... api v2.0 check Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-16 23:01:45 +03:00
Aliaksandr Valialkin	938b3b7ed1	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin	7f96712b38	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin	de1c07b937	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin	bf6d523bef	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin	d2e917d1cb	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin	a93c62cd60	lib/promscrape: fix tests after 71ea4935de	2020-10-08 19:32:48 +03:00
Aliaksandr Valialkin	0d44e371f3	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin	ad0950f630	lib/backup/actions: improve logging to be more clear to humans	2020-10-08 14:23:00 +03:00
Aliaksandr Valialkin	b51fa16177	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin	54ff78c6c9	lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does ... Such timestamps are rounded to seconds like Carbon does. See b0ba62a62d/lib/carbon/protocols.py (L197)	2020-10-06 11:38:35 +03:00
Aliaksandr Valialkin	1da41177a8	lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human	2020-10-06 11:20:36 +03:00
Aliaksandr Valialkin	97b836a6f4	lib/fs: fix GOOS=openbsd build by adding fadviseSequentialRead implementation. ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785	2020-10-05 23:32:28 +03:00
Aliaksandr Valialkin	50316070d6	lib/promscrape/discovery/openstack: code prettifying after cbe3cf683b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-05 18:12:31 +03:00
Nikolay Khramchikhin	b4c77fc6d2	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin	7505e4f390	lib/promrelabel: make a copy of label with new name for action: labelmap in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812	2020-10-05 16:20:22 +03:00
Aliaksandr Valialkin	9a7c863bd8	lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807	2020-10-05 15:19:11 +03:00
Aliaksandr Valialkin	11ddb79aeb	lib/decimal: add tests for negative values passed to maxUpExponent	2020-10-05 14:57:03 +03:00
Aliaksandr Valialkin	d448e07546	lib/decimal: properly calibrate scale for blocks with Inf values ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805	2020-10-05 14:52:59 +03:00
Aliaksandr Valialkin	fd7dd5064a	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:12:04 +03:00
Aliaksandr Valialkin	3ad7566a87	lib/storage: imrpove cache effectiveness for time series ids matching the given filters ... Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day. This should reduce CPU usage in the following cases: * when querying recently added data with small churn rate for time series * when querying historical data	2020-10-01 14:39:46 +03:00
Aliaksandr Valialkin	7c2e4e267a	lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space if there are multiple partitions with deferred merges due to disk space shortage	2020-09-29 22:53:34 +03:00
Aliaksandr Valialkin	097a4c10dd	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin	5dca7bbe85	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin	5e998e597a	lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online ... The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309	2020-09-29 13:55:53 +03:00
Aliaksandr Valialkin	338a53ccf9	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:37 +03:00
Aliaksandr Valialkin	ef416c72c2	lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm	2020-09-29 12:42:25 +03:00
Aliaksandr Valialkin	0b0259c42c	lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic	2020-09-29 12:29:50 +03:00
Aliaksandr Valialkin	8f25206d8c	lib/protoparser/prometheus: fix TestParseStream after 124f78857b	2020-09-29 12:12:33 +03:00
Aliaksandr Valialkin	81cdf2fa14	lib/{fs,filestream}: small consistency-related updates after cc90a548b1	2020-09-29 00:43:20 +03:00
Nikolay Khramchikhin	658a05ef0f	added openbsd implementations (#790) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:43:19 +03:00
Aliaksandr Valialkin	1481d6d8ff	lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer	2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin	8df33bd5c1	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin	7072db75cb	lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server ... This should reduce syscall overhead when reading big amounts of data	2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin	6d8c23fdbd	app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native ... This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics	2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin	aadbd014ff	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin	00ec2b7189	lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream ... Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.	2020-09-26 04:22:06 +03:00
Aliaksandr Valialkin	533bf76a12	lib/storage: correctly use maxBlockSize in various checks ... Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.	2020-09-24 18:13:15 +03:00
Aliaksandr Valialkin	543f3aea97	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin	90d2549428	lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue	2020-09-23 02:17:53 +03:00
Aliaksandr Valialkin	89d652b583	lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728	2020-09-22 23:27:26 +03:00
Aliaksandr Valialkin	31e341371b	lib/storage: code prettifying after be5e1222f3 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-09-22 00:42:20 +03:00
faceair	ad41e39350	add filter to getMetricIDs (#783) ... * add getMetricIDs filter * check nil filter before use	2020-09-22 00:42:19 +03:00
Aliaksandr Valialkin	d32c3f747c	lib/logger: add -loggerDisableTimestamps command-line flag for disabling timestamps in logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778	2020-09-21 19:25:50 +03:00
Aliaksandr Valialkin	f961838290	lib/promscrape/discovery/ec2: code prettifying after 312fead9a2	2020-09-21 18:44:04 +03:00
Nikolay Khramchikhin	0069353d5e	Add improvements to ec2_sd_discovery (#775) ... * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin	a9321f6a60	lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges	2020-09-21 15:51:26 +03:00
Aliaksandr Valialkin	604e8f6114	lib/decimal: optimize maxUpExponent() by eliminating division from hot path	2020-09-19 13:49:43 +03:00
Aliaksandr Valialkin	df547bf345	lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush	2020-09-19 12:51:46 +03:00
Aliaksandr Valialkin	778ea183ca	lib/decimal: properly store Inf values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-18 19:08:53 +03:00
Aliaksandr Valialkin	adb4d3b75c	lib/persistentqueue: flush data to disk every second ... Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-18 13:06:03 +03:00
Aliaksandr Valialkin	8fd791d399	lib/promscrape: avoid copying response body when scraping targets. ... This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:06:00 +03:00
Aliaksandr Valialkin	d96858b921	lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions ... This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin	3abbb38254	lib/{mergeset,storage}: compare errors with errors.Is()	2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin	ddb3519e17	lib/{mergeset,storage}: code prettifying	2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin	bf826dd828	lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions	2020-09-17 01:49:08 +03:00
Aliaksandr Valialkin	406f4fe445	app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens ... Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin	9705ac5d7a	lib/persistentqueue: code simplification after d455764a6f	2020-09-16 21:14:01 +03:00
Aliaksandr Valialkin	eee6f1e56d	lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset) ... The strategy is: - Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown. - Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data. - Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown. - Skipping of broken chunk file if it cannot be read. - Fsyncing finalized chunk files. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-16 18:13:24 +03:00
Aliaksandr Valialkin	6ce52e3702	lib/protoparser/vmimport: add more testcases for invalid timestamps and values ... Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:21:53 +03:00
Aliaksandr Valialkin	cd87ca303f	lib/protoparser: report more errors for incorrect timestamps and/or values ... Previously certain errors in timestamps and/or values could be silently skipped, which could lead to samples with zero values stored in the database. Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin	5c4e111b43	lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99	2020-09-16 02:16:13 +03:00
Aliaksandr Valialkin	0c1c1b79ba	lib/promscrape: add a link to troubleshooting docs to error message when duplicate scrape target with identical labels is skipped	2020-09-15 14:16:20 +03:00
Aliaksandr Valialkin	ca08161b54	lib/promscrape: typo fix	2020-09-12 00:14:15 +03:00
Aliaksandr Valialkin	e53235ac5c	lib/promscrape: do not reset the remaining rows when pushing a part of data to remote storage during big scrapes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753 Thanks to @PerGon and @clmssz for help with debugging.	2020-09-11 23:38:17 +03:00
Aliaksandr Valialkin	114cf24b43	lib/promscrape/discovery/dns: add __meta_dns_srv_record_target and __meta_dns_srv_record_port labels ... This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases and https://github.com/prometheus/prometheus/pull/7678 .	2020-09-11 21:35:39 +03:00
Aliaksandr Valialkin	5cf5a0e8c4	lib/protoparser/common: do not read request body when parsing timestamp query arg ... This was preventing from reading data via /api/v1/prometheus/import . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin	81c05f669b	lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin	8bd5aa3516	lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus ... This improves consistency with timestamps in Prometheus text exposition format	2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin	58d3b82ae5	app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:28:31 +03:00
Nikolay Khramchikhin	af994562c8	Added endpointslices discovery to k8s api (#760) ... This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21. See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1 * Added endpointslices discovery to k8s api Started from 1.17 k8s version endpointslices is beta, it allows to query k8s api for endpoints more efficient. It presents at scrape_config.yaml as separate role for kubernetes_sd_config. kubernetes_sd_config: - role: endpointslices * fixed typos, changed EndpointConditions signature - with values instead of pointers	2020-09-11 12:24:50 +03:00
Aliaksandr Valialkin	f307e6f432	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin	f5cb213ef9	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps ... This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin	e5c8377212	lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem	2020-09-08 19:57:20 +03:00
Nikolay Khramchikhin	fb356c434b	Changed s3 configProfile flag default, (#749) ... aws sdk has complicated logic for chosing profile name and we shouldn't set it to `default` value. It leads to bugs and improper configuration. Set it to empty value by default is safe. It will be automatically set to `default` by sdk.	2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin	d2f6f96e4a	lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699	2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin	281d715060	lib/httpserver: add -http.connTimeout command-line flag for limiting the lifetime for incoming http connections ... This can be useful for balancing incoming connections among multiple services.	2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin	4fa97430d7	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats ... Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin	95ce89e7d7	lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling ... The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc , because `sw.labels` may be reset multiple times when processing big number of rows.	2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin	bc1ca4b20b	lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections	2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin	a01c56104a	lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics ... This has been broken at 7785869ccc	2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin	deff8d419a	lib/promscrape: reduce memory usage when scraping targets with millions of metrics ... This should help when scraping /federate endpoints from Prometheus instances, which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/	2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin	5f2277624a	lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy	2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin	45e770ed20	lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores ... There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.	2020-08-28 09:49:22 +03:00
Roman Khavronenko	34ef10fbcc	lib/flagutil: avoid int overflow for arch 386 (#710) ... Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32, whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with `int overflow` error.	2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin	9a77ae9d1c	lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels ... Previously targets with big number of metrics and/or labels could generated too big buffers, which then could be re-used when scraping targets with small number of metrics. This resulted in memory waste. Now big buffers are used only for targets with big number of metrics / labels, while small buffers are used for targets with small number of metrics / labels.	2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin	3ea6444219	lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket ... This should reduce the number of resizings for the returned byte buffers.	2020-08-16 22:13:38 +03:00
Roman Khavronenko	4b89da9463	lib/decimal: rename significant decimal digits to significant figures (#698) ... The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	1b5467f7fd	lib/memory: improve log message about the memory allowed to use by VictoriaMetrics	2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin	d9f7ea1c6e	lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data ... The OS should return any buffered data in the stream without the need to set the read timeout. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin	edf3ee2a9b	lib: dump compressed block contents on error during decompression ... This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222	2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin	6e863376f7	lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty ... This should reduce memory allocations and copying when the byte slice is growing.	2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin	3efa4e4e1c	lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin	c6b0547847	lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin	9d79a3a99d	lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches	2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin	b996280c65	app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag	2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin	c82a485cf6	lib/protoparser/prometheus: typo fix in error message	2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin	9e67343756	lib/promscrape: use a hint on body length instead of body capacity ... This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin	b4119bb51e	lib/promscrape: reduce memory usage when scraping big number of targets ... Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin	1724cc241e	lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes	2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin	60c7397be5	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components ... Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin	6721e47ae9	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin	59d95961b8	lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use	2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin	890cfe5b61	lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint	2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin	e3439a6cd0	lib/promscrape: optimize per-metric hash calculations ... This increases vmagent performance by up to 10% when scraping big number of metrics	2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin	4ce1368e4b	lib/storage: mention time range used in the query that led to error message ... This should improve detecting slow queries with too big time ranges	2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin	f92255e803	lib/storage: mention tag filters used in the query that led to error message ... This should improve detecting invalid or heavy queries that lead to errors.	2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin	b3d4ff7ee2	app/vmstorage: improve error logging when the request times out	2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin	e3999ac010	lib/promscrape: show real timestamp and real duration for the scape on /targets page ... Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp	2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin	c09c881264	lib/promscrape: make errcheck happy	2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin	2dfb42a8b4	lib/promscrape: export scrape_samples_added per-target metric like Prometheus does ... This metric may be useful for detecting targets with high churn rate for the exported metrics. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683	2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin	fd9f1463df	lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails ... this is expected condition, so it is better to use WARN log level for it	2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin	d4be3efc60	lib/promscrape: add a test for scrape config for blackbox exporter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684	2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin	67cacb22ac	lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary ... This makes such binaries compatible with binaries from `master` branch (aka single-node version) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677	2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin	307281e922	lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit ... This should improve data ingestion performance when heavy searches are executed See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin	dd1d59f57a	lib/storage: properly check timeouts and pace limits ... Previously they were checked on every iteration for small number of iterations	2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin	a2039b3bbc	app/vmselect: return the upper bound on the number of found time series from storage.Search.Init ... This is used by a single-node version in order to reduce memory allocations during search. See bc8381613d for details.	2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin	b690eeff53	lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x	2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin	6c0a92a1ee	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin	13f8644f8e	lib/storage: optimize prefetching metric names for the given metricIDs	2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin	f789e0fa44	lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock	2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin	a3e91c593b	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 ... This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin	76064ba9e7	Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer	2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin	8cc2e01386	lib/backup: allow using ~/.aws/config without region ... Use us-west-2 for determining bucket region.	2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin	a2aa3a60eb	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin	3149af624d	lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 ... Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this increases query latencies since more CPUs are busy with merges. While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.	2020-07-31 17:53:13 +03:00
Aliaksandr Valialkin	29bbab0ec9	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected ... The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin	96039dcb40	lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once ... Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series, while only a single increment is needed when inserting the first data point for this time series.	2020-07-30 16:17:19 +03:00
Aliaksandr Valialkin	1e067401ba	lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-29 13:12:35 +03:00
Sasasu	96bc476e53	lib/storage: metaindexRow use memroy more efficiently (#655) ... due to memory align the metaindexRow structure use 64-byte pre object. this commit changes the order of field, make metaindexRow use 56-byte pre object. Signed-off-by: Sasasu <su@sasasu.me>	2020-07-27 23:23:25 +03:00
Aliaksandr Valialkin	f26ef58137	lib/protoparser/prometheus: add a test for cassandra-exporter ... Thanks to Seva	2020-07-27 18:37:46 +03:00
Aliaksandr Valialkin	94cc677b0c	lib/storage: slightly reduce code difference between single-node and cluster versions	2020-07-24 01:18:05 +03:00
Aliaksandr Valialkin	fb3d1380ac	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin	dbf3038637	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin	b8303afcd8	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin	7d0743422b	lib/storage: properly calculate global metrics in UpdateStats()	2020-07-23 00:35:31 +03:00
Aliaksandr Valialkin	6afdcf8a20	lib/mergeset: properly calculate global metrics in UpdateStats() ... Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.	2020-07-23 00:35:29 +03:00
Aliaksandr Valialkin	23fa44e56e	lib/storage: reorder mergeBlockStreams() args in order to make them more consistent	2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin	754eac676d	lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs ... This condition may occur after the following sequence of events: 1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs. 2) All the goroutines return from Storage.AddRows. 3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body. The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal(). This may take indefinite time.	2020-07-22 21:52:42 +03:00
Aliaksandr Valialkin	a3f48e395e	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin	67be79a0bc	lib/uint64set: optimize adding items to the set via Set.AddMulti	2020-07-21 20:57:05 +03:00
Aliaksandr Valialkin	31ef39e8da	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin	be0ab4fbfe	lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted ... This is a follow-up commit after 12b16077c4 , which didn't reset the `tsidCache` in all the required places. This could result in indefinite errors like: missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time Fix this by resetting the cache inside deleteMetricIDs function.	2020-07-14 14:05:19 +03:00
Aliaksandr Valialkin	a4c96d9e6d	lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"} metric	2020-07-14 12:15:56 +03:00
Seva Poliakov	a5e713b6e0	add vm_protoparser_rows_read_total metrics to promscrape (#624) ... * add vm_protoparser_rows_read_total metrics to promscrape move vm_protoparser_rows_read_total for promscrape to better place move vm_protoparser_rows_read_total for promscrape to better place * remove possibility of infinity loop at prometheus parser	2020-07-14 12:02:25 +03:00
Roman Khavronenko	207e93b50d	lib/flagutil: specify additional description for all Array type flags (#620) ... Array type flag is now defined as `value` type in flag description when printed. This change adds additional description to every Array type flag so it would be clear what exact type is used: ``` -remoteWrite.urlRelabelConfig array Optional path to relabel config for the corresponding -remoteWrite.url Supports array of values separated by comma or specified via multiple flags. ```	2020-07-13 22:00:03 +03:00
Roman Khavronenko	605711bde5	lib/persistentqueue: add vm_persistentqueue_bytes_pending metric (#619) ... Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount of bytes in persistentqueue flushed on disk as a difference between write and read offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes` except of accounting for bytes in-memory.	2020-07-13 21:54:54 +03:00
Roman Khavronenko	a02097e657	Extend metric vm_promscrape_targets with status label (#615) ... The change to `vm_promscrape_targets` metric suppose to improve observability for `vmagent` so it will be possible to track how many targets are up or down for every specific scrape group: ``` vm_promscrape_targets{type="static_configs", status="down"} 1 vm_promscrape_targets{type="static_configs", status="up"} 2 ```	2020-07-13 21:54:53 +03:00
Aliaksandr Valialkin	6373d377ef	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin	2012e294d1	properly calculate readCalls	2020-07-10 12:01:05 +03:00
Aliaksandr Valialkin	87f8c728bf	lib/promscrape: send Accept header similar to Prometheus when scraping targets ... This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:50:06 +03:00
Aliaksandr Valialkin	7335743d57	lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS ... Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense, since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs, i.e. to GOMAXPROCS.	2020-07-08 17:34:27 +03:00
Roman Khavronenko	929ad74de6	lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607) ... The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to respect the naming standard.	2020-07-08 14:19:27 +03:00
Aliaksandr Valialkin	e401b8d527	lib/protoparser/graphite: go fmt	2020-07-08 14:13:06 +03:00
Aliaksandr Valialkin	50ecf09042	lib/protoparser/graphite: add more tests after eb45185eef ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610	2020-07-08 14:13:03 +03:00
Seva Poliakov	1ae0334e17	Fix graphite minus one timestamp (#609) ... * fix graphite -1 timestamp * format the graphite fix -1 timestamp	2020-07-08 14:13:01 +03:00
Aliaksandr Valialkin	fad008df7e	lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag	2020-07-08 13:54:13 +03:00
Aliaksandr Valialkin	fe58462bef	lib/storage: reset MetricName->TSID cache after deleting time series ... This should prevent from adding new data points to deleted time series without the need to check for the deleted time series. This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set contains big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596 Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604	2020-07-06 22:01:24 +03:00
Aliaksandr Valialkin	77bb0e6595	lib/fs: clarify description for -fs.disableMmap command-line flag	2020-07-06 14:28:57 +03:00
Aliaksandr Valialkin	0bff96fe4b	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:44:04 +03:00
Aliaksandr Valialkin	6f1d926698	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:48 +03:00
Aliaksandr Valialkin	ee03b4ccbd	lib/promscrape: prevent from too big deadline misses on scrape retries ... The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:42:09 +03:00
Aliaksandr Valialkin	dfa83a4a35	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:25 +03:00
Aliaksandr Valialkin	8bb3622e9d	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin	6ebac3ab63	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin	2361ad8ab4	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:34 +03:00