2084 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	bb68ab99fa	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin	b0e1bb517e	lib/storage: typo fix in comments after f830edc0bc	2022-08-16 13:44:45 +03:00
Aliaksandr Valialkin	f830edc0bc	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:32:40 +03:00
Aliaksandr Valialkin	c3f8481011	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin	95d36da358	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin	c4fcd9f1c5	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin	511805d88d	lib/promscrape/discovery/dns: add support for resolving MX records ... See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:32:34 +03:00
Roman Khavronenko	a0e7432e42	lib/storage: prevent excessive loops when storage is in RO (#2962) ... * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:00 +03:00
Aliaksandr Valialkin	46d7792b72	lib/promscrape: follow-up after 2c553d5a2f ... - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:46:26 +03:00
Fury	2c553d5a2f	add support to scrape multi tenant metrics (#2950) ... * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:10:18 +03:00
Roman Khavronenko	d3f13ab85b	lib/promrelabel: fix expected test result (#2957) ... follow-up after 68c4ec9472 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:47:29 +03:00
Aliaksandr Valialkin	68c4ec9472	lib/promrelabel: do not split regex into multiple lines if it contains groups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:15:26 +03:00
Aliaksandr Valialkin	892c97e350	lib/auth: follow-up after b6a6a659f4	2022-08-07 23:14:39 +03:00
Dmytro Kozlov	b6a6a659f4	lib/auth: add tests for NewToken function (#2921) ... * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:07:57 +03:00
Aliaksandr Valialkin	9fa6b25fb2	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:29 +03:00
Aliaksandr Valialkin	0ef29ceb14	lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress ... See 7e65ad3e43 and 7e1111ff14	2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin	f2816ef031	lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs ... This option was ignored since d289ecded1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin	3e8890e71b	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels ... See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin	68de1f4e4a	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default ... This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin	02de848c88	lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f	2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin	83a4abda3f	lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:26:43 +03:00
Igor Tiunov	6e5ac32fba	YC service discovery (#2923) ... * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin	d5df08e9c2	lib/mergeset: cleanup after de6dd1cd5a ... Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:23:01 +03:00
Aliaksandr Valialkin	7c99b9eaad	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:42:43 +03:00
Aliaksandr Valialkin	6b0550c023	app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin	5a4c58f9a2	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:12:29 +03:00
Aliaksandr Valialkin	78520f2702	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache ... This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:08:51 +03:00
Aliaksandr Valialkin	de6dd1cd5a	lib/mergeset: optimize mergeInmemoryBlocks() function ... Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-27 23:58:05 +03:00
Aliaksandr Valialkin	a3f5822dc2	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge ... Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-27 23:05:02 +03:00
Aliaksandr Valialkin	be1c82beb1	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:20:27 +03:00
Aliaksandr Valialkin	5c84f09762	lib/mergeset: add tests and benchmarks for commonPrefixLen function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:24:51 +03:00
Aliaksandr Valialkin	f5676123cc	lib/pushmetrics: make fmt	2022-07-26 20:40:19 +03:00
Aliaksandr Valialkin	da11056d85	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:24:24 +03:00
Aliaksandr Valialkin	ad6b3cd47d	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:36:06 +03:00
Aliaksandr Valialkin	4c2f9a1a2e	lib/promscrape: set up=0 for partially failed scrape in stream parsing mode ... This behaviour aligns with Prometheus behavior	2022-07-22 13:29:44 +03:00
Roman Khavronenko	2914ce5ca5	vmalert: remove dependency on datasource pkg from config (#2905) ... * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 10:44:55 +02:00
Aliaksandr Valialkin	4ce5875fa8	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:36:27 +03:00
Roman Khavronenko	88edb3f6cf	vmalert: allow configuring custom headers per group (#2901) ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 15:59:55 +02:00
Aliaksandr Valialkin	0fd86e2364	lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:07 +03:00
Boris Petersen	2f9668eba5	fix assume role when running in ECS. (#2876) ... This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:33:52 +03:00
Aliaksandr Valialkin	814bb1685f	all: fix other typos in the same way as 6f4d9b2a48 does	2022-07-18 12:08:15 +03:00
zhenyuxie	f3ea7823f3	fix inmemoryBlock's Less method (#2881)	2022-07-18 11:56:17 +03:00
Nikolay	7301aa678c	lib/promscrape: adds azure service discovery (#2743) ... * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:43:18 +03:00
guidao	91faa152a5	add next retention metric (#2863) ... Co-authored-by: wangfeng <wangfeng@zhihu.com>	2022-07-13 12:37:04 +03:00
Dmytro Kozlov	306ec10c39	lib/mergeset: fix linter error (#2864)	2022-07-13 12:31:35 +03:00
Aliaksandr Valialkin	17b5ac1608	lib/mergeset: optimize merge speed a bit ... Use heap.Fix instead of heap.Pop + heap.Push when merging blocks	2022-07-12 12:50:26 +03:00
Aliaksandr Valialkin	5c8eee26bf	all: make fmt via the upcoming Go1.19	2022-07-11 19:22:15 +03:00
Aliaksandr Valialkin	f97355d9fb	lib/promscrape: properly set Host header when sending requests via http proxy ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin	10cb67adb5	app/{vmagent,vminsert}: follow-up after d19e46de55 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:30:58 +03:00
Aliaksandr Valialkin	01f55bc66b	lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin	b186b63e07	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/10759	2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin	e6ba2af7a1	lib/promscrape: fix a test after c66f676f3b	2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin	c66f676f3b	lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target ... See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin	77cbbacfdb	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics ... This reduces the number of args to vmselectapi calls	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	e1b8059086	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	a60e03b3a7	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() ... This improves the API consistency	2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin	edc76286ac	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index ... Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit 55e7afae3a, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:54:03 +03:00
Aliaksandr Valialkin	855436efd2	lib/promauth: refactor NewConfig in order to improve maintainability ... 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin	c392d6d173	app/vmagent/remotewrite: add -remoteWrite.header command-line flag for setting additional http headers to send to -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:23 +03:00
Aliaksandr Valialkin	e40b40afe6	Revert "lib/promscrape, vmagent: fix path to files (#2801)" ... This reverts commit 0a8e35835c. Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005	2022-06-30 18:23:56 +03:00
Aliaksandr Valialkin	3e2dd85f7d	all: readability improvements for query traces ... - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:20:33 +03:00
Dmytro Kozlov	0a8e35835c	lib/promscrape, vmagent: fix path to files (#2801) ... vmagent: respect `-pathPrefix` flag for static files and links	2022-06-30 16:22:54 +02:00
ttyv	bdf9f4669a	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786) ... Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin	a350d1e81c	lib/storage: return marshaled metric names from SearchMetricNames ... Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin	2c836bd398	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:50:53 +03:00
Aliaksandr Valialkin	e578549b8a	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin	a963b2a0aa	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	ba514284f1	lib/storage: add querytracer to more contexts ... querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	134751e43e	all: locate throttled loggers via logger.WithThrottler() only once and then use them ... This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 13:45:50 +03:00
Aliaksandr Valialkin	52eadb729e	lib/promscrape: always send stale markers with the real scrape timestamp ... This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:34:18 +03:00
Aliaksandr Valialkin	1c4f67c5d2	lib/promauth: add ability to send additional http headers in requests to scrape targets ... This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin	e6ed92529b	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package ... This package already has the same name, so there is no need in explicit name	2022-06-21 20:23:32 +03:00
Loki's Wager	ac411be904	BugFix part_header.go (#2763) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:56:41 +03:00
Aliaksandr Valialkin	49586566a3	docs: follow-up after e4d6b750f6	2022-06-20 17:14:43 +03:00
Nikolay	e4d6b750f6	lib/httpserver: adds flagsAuthKey command-line flag (#2758) ... * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:09:32 +03:00
Aliaksandr Valialkin	b958fc7846	lib/storage: properly take into account already registered series when -storage.maxHourlySeries or -storage.maxDailySeries limits are enabled ... The commit 5fb45173ae takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:47:47 +03:00
Aliaksandr Valialkin	55e7afae3a	lib/storage: create per-day indexes together with global indexes when registering new time series ... Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:42:10 +03:00
Aliaksandr Valialkin	5fb45173ae	lib/storage: do not register new series if -storage.maxHourlySeries or -storage.maxDailySeries limits are exceeded ... Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin	62e2371a67	lib/storage: reset metric id caches for the previous and the current hour ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin	c18f8cccfa	lib/promrelabel: support action: graphite relabeling ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:24:22 +03:00
Aliaksandr Valialkin	ec7963208d	app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb ... This allows filling the seriesCountByFocusLabelValue list in the /api/v1/status/tsdb response with label values for the specified focusLabel, which contain the highest number of time series. TODO: add this to Cardinality explorer at VMUI - https://docs.victoriametrics.com/#cardinality-explorer	2022-06-14 18:36:54 +03:00
Aliaksandr Valialkin	b6c1ca12b7	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:32:38 +03:00
Aliaksandr Valialkin	a75e59700f	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:27:59 +03:00
Aliaksandr Valialkin	52cf05c6d2	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:27:40 +03:00
Aliaksandr Valialkin	374beb350e	app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them	2022-06-12 04:32:13 +03:00
Aliaksandr Valialkin	2bcb960f17	all: improve query tracing coverage for indexdb search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:07:07 +03:00
Howie	76f05f8670	feat: rule limit (#2676) ... vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 08:21:30 +02:00
Aliaksandr Valialkin	12ac255dae	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:06:52 +03:00
Dmytro Kozlov	018d2303c4	Cardinality explorer (#2625) ... * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:43:05 +03:00
Roman Khavronenko	63b538ecd1	vmagent: update SD duration histogram metric if SD is active (#2677) ... The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:46:44 +03:00
Roman Khavronenko	1ee1e986da	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 14:55:09 +03:00
Aliaksandr Valialkin	a5814fe16a	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string ... This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:22:07 +03:00
Aliaksandr Valialkin	8608dd093c	all: follow-up after 8edb390e21 ... - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 00:57:09 +03:00
Aliaksandr Valialkin	6f0a0e3072	lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534 ... - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:40:52 +03:00
Aliaksandr Valialkin	cfefdde042	lib/promauth: follow-up after 006b8c7534 ... - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:01:16 +03:00
Aliaksandr Valialkin	0922ed2b7e	lib/promscrape: add -promscrape.cluster.name command-line flag ... This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 00:37:01 +03:00
Dmytro Kozlov	8edb390e21	lib/promscrape: adds service discovery visualization for /targets page(#2675) ... * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-03 15:38:45 +02:00
Nikolay	a18914abee	lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 (#2672)	2022-06-01 20:44:45 +02:00
hadesy	006b8c7534	promscrape/discovery: support kubeconfig (#2533)	2022-06-01 20:34:00 +02:00
Aliaksandr Valialkin	ca689fec54	docs/CHANGELOG.md: follow-up after 2177089f94	2022-06-01 14:51:26 +03:00
Aliaksandr Valialkin	ea06d2fd3c	lib/storage: stop background merge when storage enters read-only mode ... This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:36:45 +03:00
Roman Khavronenko	642eb1c534	lib/storage: make indexdb/tagFilters cache size configurable (#2667) ... The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 10:07:53 +02:00
Roman Khavronenko	2177089f94	promrelabel: add support of lowercase and uppercase relabeling actions (#2665) ... * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 10:02:37 +02:00
Aliaksandr Valialkin	41958ed5dd	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin	d2567ccdd6	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum ... In this case there is no need in converting int64 to int	2022-06-01 01:42:34 +03:00
Aliaksandr Valialkin	a1add5c2c7	lib/storage: make fmt	2022-05-31 12:54:37 +03:00
Aliaksandr Valialkin	bac75ea8a2	lib/storage: do not take into account series from the next day when match[] filter is passed to /api/v1/status/tsdb	2022-05-31 12:15:26 +03:00
Dmytro Kozlov	11f91532c5	issue-2594: use embedded for static files (#2650) ... embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 01:55:28 +02:00
Dmytro Kozlov	1eb29794e6	removed redundant return (fixed linter) (#2647) ... * removed redundant return * updated lint package version	2022-05-26 16:24:01 +02:00
Aliaksandr Valialkin	796804e4b0	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag ... This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 22:59:36 +03:00
Aliaksandr Valialkin	f6d11a49aa	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag ... This is a follow-up for 0fbf59199a See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:05:29 +03:00
阳明	0fbf59199a	lib/storage: Remove the effect of time zone on next retention period (#2568) (#2574)	2022-05-25 15:08:24 +03:00
Roman Khavronenko	d5eb6afe26	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) ... * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:01:37 +03:00
Boris Petersen	3df8caca15	Add ability to sign requests for all AWS services (#2604) ... This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-18 14:58:31 +02:00
Aliaksandr Valialkin	a0727ab1b1	docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag	2022-05-12 18:50:29 +03:00
Aliaksandr Valialkin	9ea3f0c0d3	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future ... Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:33:22 +03:00
Aliaksandr Valialkin	123aa4c79e	lib/promscrape: properly implement ScrapeConfig.clone() ... Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit 67b10896d2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:05:40 +03:00
Aliaksandr Valialkin	1dc4cc243b	lib/promscrape: rename promscrape_stale_samples_created_total metric to vm_promscrape_stale_samples_created_total, so its name is consistent with the rest of vm_promscrape_ metrics	2022-05-06 15:33:13 +03:00
Aliaksandr Valialkin	d5b55fe22d	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section	2022-05-06 12:43:29 +03:00
Aliaksandr Valialkin	97f9c2f667	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call ... Previously filters wheren't passed to this call after the commit 0e09fdb8b0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:00:23 +03:00
Aliaksandr Valialkin	d285c2fea7	lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query ... The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: 0e09fdb8b0 d289ecded1	2022-05-05 10:29:34 +03:00
Dmytro Kozlov	7dd9f3b98e	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535) ... * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:03 +03:00
Nikolay	d289ecded1	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458) ... * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:24:19 +03:00
Nikolay	3575aabeaf	lib/promscrape: adds correct http status codes for redirect (#2530) ... standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 13:31:31 +03:00
Aliaksandr Valialkin	0d86644d65	lib/storage: leave the last sample per each discrete interval during the deduplicaton ... This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:50:45 +03:00
Artem Navoiev	37cf509c3a	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487) ... * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:00:15 +03:00
Aliaksandr Valialkin	20bc2a2c44	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:56:39 +03:00
Dima Lazerka	84683b8569	Fix targetstatus qtpl paths (#2517) ... Ran `make quicktemplate-gen` from the root directory	2022-04-29 10:36:03 +03:00
Aliaksandr Valialkin	6be07e8c25	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing ... This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:26:01 +03:00
Aliaksandr Valialkin	9fe1bf5d53	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:23:19 +03:00
Aliaksandr Valialkin	eb5d7ad089	lib/promauth: add support for min_version option at tls_config section in the same way as Prometheus does	2022-04-23 00:16:39 +03:00
Aliaksandr Valialkin	174431e31b	lib/promauth: add support for proxy_url option at oauth2 section in the same way as Prometheus does	2022-04-23 00:00:44 +03:00
Aliaksandr Valialkin	18b14aad8e	lib/promauth: add support for tls_config section at oauth2 config in the same way as Prometheus does	2022-04-22 23:51:07 +03:00
Aliaksandr Valialkin	6f79b2b68b	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects ... Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:14:17 +03:00
Aliaksandr Valialkin	15190fcdae	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does ... See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:41 +03:00
Aliaksandr Valialkin	57a0aa204d	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems ... Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:22:01 +03:00
Aliaksandr Valialkin	67b10896d2	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated ... This is a follow-up after 26b78ad707	2022-04-22 13:19:43 +03:00
Aliaksandr Valialkin	98129d4a8e	app/vmstorage: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page ... These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:18:39 +03:00
Aliaksandr Valialkin	167d1bea8f	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for 2341bd48d7	2022-04-21 13:06:22 +03:00
Aliaksandr Valialkin	c75d0095f5	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:40:53 +03:00
Aliaksandr Valialkin	82e34984dd	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:33 +03:00
Aliaksandr Valialkin	a2de31f8d3	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects ... There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:40:25 +03:00
Aliaksandr Valialkin	2341bd48d7	lib/promscrape: follow-up after 91e290a8ff	2022-04-20 16:11:37 +03:00
Nikolay	91e290a8ff	lib/promscrape: reduce latency for k8s GetLabels (#2454) ... replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:09:40 +03:00
Aliaksandr Valialkin	3d0549c982	lib/promscrape: optimize getScrapeWork() function ... Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:37:00 +03:00
Aliaksandr Valialkin	4513893ead	lib/promscrape: use a hash over target labels as a key for dropped targets' map ... This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:37:00 +03:00
Dmytro Kozlov	136a44bcfc	lib/promscrape: simply update UI (#2479) ... * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 10:25:04 +02:00
Aliaksandr Valialkin	f6d0e5e74a	all: typo fix: Kuberntes -> Kubernetes	2022-04-20 10:50:49 +03:00
Dmytro Kozlov	a3ee275149	lib/promscrape: Enable filters for endpoint and labels (#2466) ... * lib/promscrape: Enable filters for endpoint and labels * lib/promscrape: cleanup * lib/promscrape: update template * lib/promscrape: move logic filter logic to backend * lib/promscrape: updated placeholder * lib/promscrape: updated placeholder * lib/promscrape: use two different fields for filters, updated form, added error on parsing queries * lib/promscrape: rename functions * lib/promscrape: removed unused values * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-19 18:26:21 +03:00
Nikolay	26b78ad707	lib/promscrape: adds job restart method (#2455) ... * lib/promscrape: adds job restart method it must restart only ScrapeConfig with changed content this change greatly reduce time, that needed for job restart and it should decrease possible data loss when config frequently changed at kubernetes based deployments Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 20:28:46 +03:00
Aliaksandr Valialkin	1097ebebe6	lib/httpserver: clarify that -tls flag enables TLS for http requests to -httpListenAddr	2022-04-16 16:59:26 +03:00
Aliaksandr Valialkin	cad488fe7e	app/vmstorage: add support for mTLS cipher suites via -cluster.tlsCipherSuites command-line flag ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404	2022-04-16 16:39:21 +03:00
Aliaksandr Valialkin	7810375c5f	lib/httpserver: move the code, which creates tls.Config, into lib/netutil/tls.go ... This syncs the corresponding code with cluster branch	2022-04-16 15:52:36 +03:00
Aliaksandr Valialkin	7e4bdf31ba	lib/httpserver: follow up after def0032c7d	2022-04-16 15:27:21 +03:00
Dmytro Kozlov	def0032c7d	lib/httpserver: added tlsCipherSuites flag (#2468) ... * lib/httpserver: added tlsCipherSuites flag * lib/httpserver: compare lower case strings * lib/httpserver: use EqualFold * lib/httpserver: used flagutil.NewArray, supported only strings cipher suites * lib/httpserver: updated flag description, added flag to documentation * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 15:07:07 +03:00
Aliaksandr Valialkin	ebaa1c7ad5	lib/promscrape: follow-up after baa1c24b36	2022-04-16 14:25:54 +03:00
Nikolay	baa1c24b36	lib/promscrape: removes omitempty for ScrapeConfig (#2457) ... This change fixes incorrect marshalling for ScrapeConfig it affects http endpoint and ScrapeConfig checksum. With omitempty, custom Marshaller is not called if field is not a pointer. Previously this issue happened at vmalert	2022-04-16 13:22:11 +03:00
Aliaksandr Valialkin	c6eb404c69	lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* ... This allows Go complier to generate more optimal code without bound checks	2022-04-12 12:55:21 +03:00
Aliaksandr Valialkin	f3d4671bb6	lib/promscrape: follow-up after 7e79adfb55	2022-04-12 12:36:17 +03:00
Nikolay	7e79adfb55	lib/promscrape: allows to use k8s pod name as clusterMemberNum (#2436) ... * lib/promscrape: allows to use k8s pod name as clusterMemberNum it must improve user expirience and simplify clustering scrapers. it must allow to use vmagent cluster with distroless images https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-12 12:24:11 +03:00
Aliaksandr Valialkin	54de0531a4	app/vmstorage: properly handle maxSeries limit passed from vmselect to vmstorage	2022-04-12 11:23:04 +03:00
Aliaksandr Valialkin	deaa8c1ffa	lib/protoparser/native: follow-up after fe01f4803d	2022-04-11 19:27:07 +03:00
Nikolay	fe01f4803d	lib/protoparser/native: fixes parseStream dead-lock (#2423) ... previously, if native block cannot be unmarshaled, wg.Done wasn't called by unmarshal work. It leads to connection blocking and possible dead-lock at client side	2022-04-11 19:22:24 +03:00
Aliaksandr Valialkin	a96eb16329	lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to ... This metric is equivalent to `vm_available_memory_bytes`, but it has better name, since the metric is related to a process, not VictoriaMetrics itself. Leave `vm_available_memory_bytes` for backwards compatibility.	2022-04-07 15:23:00 +03:00
Aliaksandr Valialkin	57143e9435	lib/storage: increase the number of rawRowsShard shards on systems with more than 4 CPU cores ... This should improve data ingestion scalability on systems with many CPU cores	2022-04-06 19:49:20 +03:00
Aliaksandr Valialkin	7bad7133bc	lib/mergeset: use more rawItemsShard shards on multi-CPU systems ... This should improve the scalability for registering of new time series on multi-CPU system	2022-04-06 19:35:55 +03:00
Aliaksandr Valialkin	ad35068c3a	lib/mergeset: skip common prefixes when comparing inmemoryBlock items ... This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData if they have common prefix. While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items. This should improve performance for inmemoryBlock.MarshalSortedData during background merge.	2022-04-06 18:51:36 +03:00
Aliaksandr Valialkin	5acd70109b	lib/protoparser: remove superflowous memory allocations during protocol parsing	2022-04-06 14:00:08 +03:00
Aliaksandr Valialkin	50cf74ce4b	lib/storage: reuse sync.WaitGroup objects ... This reduces GC load by up to 10% according to memory profiling	2022-04-06 13:34:04 +03:00
Aliaksandr Valialkin	077193d87c	lib/cgroup: reduce the default GOGC value from 50% to 30% ... This reduces memory usage under production workloads by up to 10%, while CPU spent on GC remains roughly the same. The CPU spent on GC can be monitored with go_memstats_gc_cpu_fraction metric	2022-04-06 13:32:07 +03:00
Aliaksandr Valialkin	319e910897	lib/workingsetcache: reuse prev cache after its reset ... This should reduce memory churn rate	2022-04-05 20:37:45 +03:00
Aliaksandr Valialkin	29cebb3d95	lib/workingsetcache: check more frequently for cache size overflow ... This should reduce the probability of cache size limit overflow	2022-04-05 18:05:43 +03:00
Aliaksandr Valialkin	4785d04312	lib/workingsetcache: reduce the expiration duration from 20 minutes to 10 minutes ... This should reduce memory usage for the cache under high churn rate	2022-04-05 17:12:13 +03:00
Nikolay	0c0efc7781	vmctl verify-blocks command (#2390) ... * lib/protoparser: changes ParseStream for native format uses reader instead of http.Request updates app/vmagent and app/vmagent method usage * app/vmctl: add verify-block subcommand it allows to check exported from VictoriaMetrics data block in native format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362 Update app/vmctl/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-04-05 16:01:32 +02:00
Nikolay	9a88c1a91e	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293) ... * lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache It should decrease memory usage for regexp caching with storing cacheEntry by pointer - golang map should be able to effectivly shrink it's size original issue with this case - unexpected map grows and storage OOM Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Adds missing metrics for regexp cache and regexpPrefixes cache * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 12:54:50 +02:00
Aliaksandr Valialkin	6e364e19ef	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin	e3a10b327c	lib/blockcache: properly remove references to deleted parts ... Previously references to deleted parts may remain active as cache.m keys. This could prevent from proper memory de-allocation. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This is a follow-up for 88605a7ea2	2022-03-18 17:07:59 +02:00
Aliaksandr Valialkin	2ae3a9a8a3	lib/storage: reduce the interval for checking for free disk space from 30 seconds to 1 second ... This should reduce the probability of out of disk space panics when -storage.minFreeDiskSpaceBytes is set to low values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2305	2022-03-18 16:52:27 +02:00
Aliaksandr Valialkin	88605a7ea2	lib/blockcache: properly release memory occupied by deleted entries ... Proviously the deleted entries could remain referenced via lastAccessHeap for long time. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-03-18 16:52:27 +02:00
jduncan0000	e5868b9c29	Fix for issue #2255 - matchTagFilters for positive empty-match filters (#2304) ... * fix for issue 2255 - matchTagFilters for positive empty-match filters * add example to comments * formatting * add test for positive empty match * formatting	2022-03-18 12:58:22 +02:00
Aliaksandr Valialkin	3eef1ddc7d	lib/storage: trashing -> thrashing typo in docs ... This is a follow-up for 918ed5cb32	2022-03-16 13:05:26 +02:00
Vic (Shihang) Li	918ed5cb32	fix: change thrashing typo (#2317)	2022-03-15 07:05:52 +00:00
Aliaksandr Valialkin	0a4aadffac	lib/mergeset: remove aux buffers from inmemoryPart ... This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:08:44 +02:00
Aliaksandr Valialkin	c84a8b34cc	lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock ... This should improve performance when registering new time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 16:46:37 +02:00
Aliaksandr Valialkin	7da4068f48	lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars	2022-03-03 15:48:22 +02:00
Aliaksandr Valialkin	e8fdb27625	lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool ... The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart, so sync.Pool usage should reduce overall memory usage and improve performance because of better locality of reference when marshaling inmemoryBlock to inmemoryPart. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 15:44:02 +02:00
Aliaksandr Valialkin	59877d9f32	lib/{mergeset,storage}: tune compression levels for small blocks ... This should reduce CPU usage spent on compression	2022-02-25 15:33:40 +02:00
Aliaksandr Valialkin	7e99bbb967	lib/storage: document why job-like and instance-like labels must be stored at mn.Tags[0] and mn.Tags[1] ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2244	2022-02-25 13:21:07 +02:00
Aliaksandr Valialkin	8bf3fb917a	lib/storage: add a comment to indexSearch.containsTimeRange() on why it allows false positives ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2239	2022-02-24 12:47:27 +02:00
Aliaksandr Valialkin	a16f1ae565	lib/storage: properly handle series selector matching multiple metric names plus a negative filter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238 This is a follow-up for 00cbb099b6	2022-02-24 12:15:54 +02:00
Aliaksandr Valialkin	af5bdb9254	lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort() ... There is no need to sort the underlying data according to sorted items there. This should reduce cpu usage when registering new time series in `indexdb`. Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245	2022-02-24 11:20:32 +02:00
Aliaksandr Valialkin	3f49bdaeff	lib/promrelabel: add support for conditional relabeling via if filter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998	2022-02-24 02:27:26 +02:00
Aliaksandr Valialkin	d128a5bf99	lib/workingsetcache: do not rotate cache if it is in whole state ... This should reduce the maximum memory usage for the cache in `whole` state	2022-02-23 22:55:18 +02:00
Aliaksandr Valialkin	62b46007c5	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin	f72b35665f	lib/storage: optimize /api/v1/status/tsdb call by skipping all the artificially created tag entries at once ... This is a follow-up for b71be42d90	2022-02-21 18:23:35 +02:00
Aliaksandr Valialkin	ed12c60826	lib/mergeset: typo fix after b6ed9afd6d	2022-02-21 17:58:22 +02:00
Aliaksandr Valialkin	5d45ea1003	lib/blockcache: evict entries from the cache in LRU order ... This should improve hit rate for smaller caches	2022-02-21 17:44:24 +02:00
Roman Khavronenko	69d1893f4c	Consul SD - update services on the watcher's start (#2202) ... * lib/discovery/consul: update services on the watcher's start Previously, watcher's start was only initing goroutines for discovery but not waiting for the first iteration to end. It means first Consul discovery wasn't returning discovered targets until the next iteration. The change makes the watcher's start blocking until we get first discovery iteration done and all registries updated. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: remove workarounds for consul SD Now when consul SD lib properly updates services on the first start, we don't need workarounds in vmalert. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/discovery/consul: update after review Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:32:45 +02:00
Roman Khavronenko	b6ed9afd6d	lib: allow to configure cache size by type (#2206) ... * lib: allow to configure cache size by type https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 13:50:34 +02:00
Aliaksandr Valialkin	2b87b4d183	lib/storage: typo fix after c3affb0c4f	2022-02-17 12:55:54 +02:00
Aliaksandr Valialkin	c3affb0c4f	lib/storage: simplify code for searching for label values ... This is a follow-up after 9dd191b27c	2022-02-17 12:29:38 +02:00
Aliaksandr Valialkin	9dd191b27c	lib/storage: properly skip composite tag entries when searching for tag names or tag values ... This is a follow-up for b71be42d90 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 23:01:19 +02:00
Aliaksandr Valialkin	5366d9be73	lib/blockcache: fix TestCache by ensuring that the cache size can be divided by the number of cache shards ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2204	2022-02-16 18:47:35 +02:00
Aliaksandr Valialkin	6ff71474a6	lib/storage: document why tsid cache is reset before saving it to disk ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2205	2022-02-16 18:37:56 +02:00
Aliaksandr Valialkin	b71be42d90	lib/storage: use binary search instead of full scan for skipping artificial tags when searching for tag names or tag values ... This should improve performance for /api/v1/labels and /api/v1/label/<label_name>/values See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 18:15:41 +02:00
Roman Khavronenko	d91c1d4eee	vmagent: fix js error on CollapseAll/ExpandAll buttons click (#2192) ... * vmagent: fix js error on CollapseAll/ExpandAll buttons click `Uncaught TypeError: Cannot read properties of null (reading 'style')` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-15 12:52:48 +02:00
Corporte Gadfly	ad6bdd78d0	match fileSDCheckInterval with prometheus file_sd_config default (#2188)	2022-02-15 12:04:26 +02:00
Aliaksandr Valialkin	1215f51043	docs/CHANGELOG.md: document 3d890e89f1	2022-02-14 17:39:12 +02:00
Nikolay	3d890e89f1	Adds server certificate reload for lib/http (#2186) ... * Adds server certificate reload for lib/http https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171 * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-14 17:32:13 +02:00
Nikolay	c90c1c4d54	fixes all_tenants query option usage for openstack service discovery (#2184) ... explicit use configuration parametr instead of conditional add https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182	2022-02-14 13:07:30 +02:00
Aliaksandr Valialkin	f10c38b827	lib/promscrape: add expand all and collapse all buttons to /targets page	2022-02-12 18:41:29 +02:00
Aliaksandr Valialkin	96dce63dbd	lib/storage: tune the logic for pre-populating of the per-day inverted index for the next day ... - Postpone the pre-poulation to the last hour of the current day. This should reduce the number of useless entries in the next per-day index, which shouldn't be created there, when the corresponding time series are stopped to be pushed during the current day. - Make the pre-population more smooth in time by using the hash of MetricID instead of MetricID itself when calculating the need for for the given MetricID pre-population. - Sync the logic for pre-population of the next day inverted index with the logic of pre-populating tsid cache after indexdb rotation. This should improve code maintainability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401	2022-02-12 16:33:16 +02:00
artifactori	ea153e5f90	Show gce sdconfig zone on vmagent:8429/config (#2178) ... * vmagent: add test for marshalling gce sdconfig with ZoneYAML * vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig	2022-02-12 00:39:23 +02:00
Roman Khavronenko	cf1a8bce6b	lib/index: reduce read/write load after indexDB rotation (#2177) ... * lib/index: reduce read/write load after indexDB rotation IndexDB in VM is responsible for storing TSID - ID's used for identifying time series. The index is stored on disk and used by both ingestion and read path. IndexDB is stored separately to data parts and is global for all stored data. It can't be deleted partially as VM deletes data parts. Instead, indexDB is rotated once in `retention` interval. The rotation procedure means that `current` indexDB becomes `previous`, and new freshly created indexDB struct becomes `current`. So in any time, VM holds indexDB for current and previous retention periods. When time series is ingested or queried, VM checks if its TSID is present in `current` indexDB. If it is missing, it checks the `previous` indexDB. If TSID was found, it gets copied to the `current` indexDB. In this way `current` indexDB stores only series which were active during the retention period. To improve indexDB lookups, VM uses a cache layer called `tsidCache`. Both write and read path consult `tsidCache` and on miss the relad lookup happens. When rotation happens, VM resets the `tsidCache`. This is needed for ingestion path to trigger `current` indexDB re-population. Since index re-population requires additional resources, every index rotation event may cause some extra load on CPU and disk. While it may be unnoticeable for most of the cases, for systems with very high number of unique series each rotation may lead to performance degradation for some period of time. This PR makes an attempt to smooth out resource usage after the rotation. The changes are following: 1. `tsidCache` is no longer reset after the rotation; 2. Instead, each entry in `tsidCache` gains a notion of indexDB to which they belong; 3. On ingestion path after the rotation we check if requested TSID was found in `tsidCache`. Then we have 3 branches: 3.1 Fast path. It was found, and belongs to the `current` indexDB. Return TSID. 3.2 Slow path. It wasn't found, so we generate it from scratch, add to `current` indexDB, add it to `tsidCache`. 3.3 Smooth path. It was found but does not belong to the `current` indexDB. In this case, we add it to the `current` indexDB with some probability. The probability is based on time passed since the last rotation with some threshold. The more time has passed since rotation the higher is chance to re-populate `current` indexDB. The default re-population interval in this PR is set to `1h`, during which entries from `previous` index supposed to slowly re-populate `current` index. The new metric `vm_timeseries_repopulated_total` was added to identify how many TSIDs were moved from `previous` indexDB to the `current` indexDB. This metric supposed to grow only during the first `1h` after the last rotation. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-12 00:30:08 +02:00
Aliaksandr Valialkin	08428464e9	lib/storage: fix broken BenchmarkHeadPostingForMatchers for {i=~".*"} after f4dead529f ... The commit f4dead529f makes such query to return nothing instead of all the time series. This aligns more with Prometheus behaviour.	2022-02-12 00:27:10 +02:00
Roman Khavronenko	e3adcbec6e	lib/promscrape: support prometheus-like duration in scrape configs (#2169) ... * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:00 +02:00
Aliaksandr Valialkin	3cb72ccc2a	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpointslice_{label,annotation}* labels to be consistent with other role values for Kubernetes service discovery	2022-02-11 14:54:47 +02:00
Nikolay	4e7f7f3302	fixes service discovery for kubernetes (#2173) ... * fixes service discovery for kubernetes now it must take in account all pods that belong to the discovered endpoint and endpointslice adds simple test for endpoints https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134 * wip * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 13:34:22 +02:00
Aliaksandr Valialkin	f9a17cb5fe	lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:09:46 +02:00
Aliaksandr Valialkin	a9bb22b213	lib/blockcache: use higher number of shards for higher number of CPU cores ... This should reduce mutex contention and increase performance Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:06:12 +02:00
Aliaksandr Valialkin	db8c4054e5	lib/promscrape: fix errors in test config ... The errors were discovered after enabling strict parse mode by default. See 9bb60ab00f	2022-02-08 19:56:37 +02:00
Aliaksandr Valialkin	4507b111a9	lib/blockcache: split the cache into multiple shards ... This should reduce contention on cache mutex on hosts with many CPU cores, which, in turn, should increase overall throughput for the cache. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-08 19:44:29 +02:00
Aliaksandr Valialkin	2455a988e4	lib/mergeset: tune sizes for indexdb/dataBlocks and indexdb/indexBlocks according to production workload ... This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742	2022-02-08 17:58:49 +02:00
Aliaksandr Valialkin	9bb60ab00f	lib/promscrape: set -promscrape.config.strictParse to true by default ... This allows detecting long-living silent errors in -promscrape.config	2022-02-08 15:41:43 +02:00
Aliaksandr Valialkin	a19e7f8c5b	lib/blockcache: make fmt	2022-02-08 15:24:11 +02:00
Aliaksandr Valialkin	d0f785defd	lib/blockcache: eliminate possible race when Cache.Put is called for the same entry from multiple goroutines ... The race could result in incorrect cache size tracking, which, in turn, could result in too frequent cache cleaning	2022-02-08 01:10:43 +02:00
Aliaksandr Valialkin	46bd2c4d6d	lib/blockcache: increase the lifetime for rarely accessed blocks from 2 minutes to 5 minutes ... This should improve data ingestion speed if time series samples are ingested with interval bigger than 2 minutes. The actual interval could exceed 2 minutes if the original interval between samples doesn't exceed 2 minutes in the case of slow inserts. Slow inserts may appear in the following cases: * Big number of new time series are pushed to VictoriaMetrics, so they couldn't be registered in 2 minutes. * MetricName->tsid cache reset on indexdb rotation or due to unclean shutdown. In this case VictoriaMetrics needs to load MetricName->tsid entries for all the incoming series from IndexDB. IndexDB uses the block cache for increasing lookup performance. If the cache has no the needed block, then IndexDB reads and unpacks the block from disk. This requires an extra disk read IO and CPU. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This also should increase performance for periodically executed queries with intervals from 2 minutes to 5 minutes. See the previous similar commit - 43103be011 It is possible that the timeout can be increased further. Let's collect production numbers for this change so the timeout could be adjusted further.	2022-02-08 00:15:56 +02:00
Aliaksandr Valialkin	e86b7cc9a5	lib/workingsetcache: use the original cache size limits when rotating caches ... Previously limits for new caches were taken from cache stats. These limits could mismatch the original limits. This could result in failed cache load if the stored cache has been created with the limits obtained from cache stats.	2022-02-08 00:10:14 +02:00
Aliaksandr Valialkin	cde4664f0d	lib/blockcache: return proper number of entries from the cache ... This has been broken in 0d7374ad2f	2022-02-07 19:28:42 +02:00
Aliaksandr Valialkin	b5b3c585b3	lib/promscrape: show the total number of scrapes and the total number of scrape errors per target at /targets page ... This information may be useful when debugging unreliable scrape targets	2022-02-03 20:22:41 +02:00
Aliaksandr Valialkin	2968779f16	lib/promscrape: provide the ability to fetch target responses on behalf of vmagent or single-node VictoriaMetrics ... This feature may be useful when debugging metrics for the given target located in isolated environment	2022-02-03 19:00:55 +02:00
Aliaksandr Valialkin	9c62b25ad6	lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 00:57:14 +02:00
Aliaksandr Valialkin	4bdd10ab90	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations ... Follow-up for f4989edd96 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin	e13ce2ee98	lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64()	2022-01-31 23:36:48 +02:00
Aliaksandr Valialkin	a8509c112a	lib/storage: avoid allocations of tsidPrev on every blockStreamReader.NextBlock() call ... This is a follow-up for 00b7c97d2a Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-31 22:46:53 +02:00
Aliaksandr Valialkin	f50cf60534	lib/cgroup: fall back to runtime.NumCPU() when determining process_cpu_cores_available metric if it is impossible to determine cpu quota via cgroups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:30:14 +02:00
Aliaksandr Valialkin	ead66155ef	lib/cgroup: expose process_cpu_cores_available metric ... This metric shows the number of CPU cores available to the process. This allows creating alerting rules on CPU saturation with the following query: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:24:41 +02:00
Aliaksandr Valialkin	96aa3761fc	lib/storage/table.go: add missing tb.ptwsLock.Unlock() before the return ... This is a follow-up for a1083d0531 See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2103	2022-01-28 14:15:42 +02:00
匠心零度	1999bbfe82	optimized code (#2103) ... * optimized code ,because only the first error,so no need var errors []error * optimized code ,because only the first error,so no need var errors []error Co-authored-by: lirenzuo <lirenzuo@shein.com>	2022-01-28 14:15:41 +02:00
Aliaksandr Valialkin	f4989edd96	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions ... Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:24:44 +02:00
Aliaksandr Valialkin	91f2af2d7a	lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock ... This should reduce the memory required for indexdb/dataBlocks cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-24 18:50:40 +02:00
Aliaksandr Valialkin	4c13bae1cf	lib/logger: removed broken test after 746ee191e8	2022-01-24 12:14:32 +02:00
Aliaksandr Valialkin	746ee191e8	lib/logger/throttler.go: show the original location of the error and warning message ... Previously the location inside LogThrottler implementation was shown. This could complicate debugging.	2022-01-23 13:55:00 +02:00
Aliaksandr Valialkin	0d7374ad2f	lib/blockcache: optimize blockcache a bit ... - Optimize Cache.RemoveBlocksFromPart(), so it doesn't need to iterate over all the cached blocks. - Cache blocks if there were no cache misses during the last 2 minutes. This may be the case when new blocks are added simultaneously to the storage and to the cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-23 13:13:45 +02:00
Aliaksandr Valialkin	ede93469ea	lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-21 12:45:43 +02:00
Aliaksandr Valialkin	5f84b17ed6	lib/storage: properly limit cardinality when ingesting multiple samples for the same time series in a single request	2022-01-21 12:38:09 +02:00
Aliaksandr Valialkin	00b7c97d2a	lib/storage: verify that blocks in a single part are sorted by TSID when reading sequential blocks from the part ... This may help narrowing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:36:37 +02:00
Aliaksandr Valialkin	ea87f21e23	lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used. ... This may help nailing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:13:14 +02:00
Aliaksandr Valialkin	9797c928ef	lib/blockcache: add missing dependency after 145337792d ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:50:44 +02:00
Aliaksandr Valialkin	145337792d	lib/{mergeset,storage}: properly limit cache sizes for indexdb ... Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`, since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded, which could result to out of memory errors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:37:17 +02:00
Aliaksandr Valialkin	1d05444b33	lib/promscrape: expose promscrape_stale_samples_created_total metric for monitoring the number of created stale samples	2022-01-14 01:00:46 +02:00
Aliaksandr Valialkin	80f03177c4	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_node_provider_id label for discovered Kubernetes nodes in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/9603	2022-01-13 23:16:02 +02:00
Aliaksandr Valialkin	355a63733d	lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace ... See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881	2022-01-13 22:44:35 +02:00
Aliaksandr Valialkin	17eb86a689	lib/promscrape/discovery/dockerswarm: follow up after 68a117a25a ... - Document the bugfix at docs/CHANGELOG.md - Set __address__ field after copying commonLabels to the resulting map of discovered labels. This makes sure that the correct __address__ label is used.	2022-01-11 09:20:10 +02:00
Alexander Shtuchkin	68a117a25a	Fix for #2038: Make correct __address__ value for dockerswarm promscrape (#2041)	2022-01-11 08:59:06 +02:00
Aliaksandr Valialkin	e4e36383e2	lib/promscrape: do not send staleness markers on graceful shutdown ... This follows Prometheus behavior. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079	2022-01-07 01:17:57 +02:00
Aliaksandr Valialkin	178dd87e26	lib/storage: follow-up for 38bf5fc136	2022-01-05 16:00:11 +02:00
weng zhao	38bf5fc136	vmstorage: fix query like {foo=~"bar|"} return extra timeseries cause by negative filter transformation malfunction (#2032) ... 1. L2749 make kb.B remain the value of comonPrefix instead of tf.prefix 2. L2762 avoid change tf.value from "bar|" to ".+r|"	2022-01-05 15:59:15 +02:00
Aliaksandr Valialkin	cbaa2af280	lib/promscrape: scrape replicated targets at different offsets in vmagent replicated clustering mode ... This guarantees that the deduplication consistently leaves samples from the same vmagent replica. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets	2021-12-23 00:20:39 +02:00
Nikolay	8ff7da7202	adds restore.lock (#1988) ... * adds restore.lock it must prevent from running storage after incomplete restore process https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 * return back flock file deletion * Apply suggestions from code review * wip * docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-22 13:10:15 +02:00
Aliaksandr Valialkin	ce333f28d8	all: use logger.WithThrottler() where appropriate	2021-12-21 17:03:25 +02:00
Roman Khavronenko	34fdc8881b	vmagent: add error log for skipped data block when rejected by receiv… (#1956) ... * vmagent: add error log for skipped data block when rejected by receiving side Previously, rejected data blocks were silently dropped - only metrics were update. From operational perspective, having an additional logging for such cases is preferable. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: throttle log messages about skipped blocks The new type of logger was added to logger pacakge. This new type supposed to control number of logged messages by time. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/logger: make LogThrottler public, so its methods can be inspected by external packages Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 16:36:09 +02:00
Aliaksandr Valialkin	b9363d9726	lib/promscrape: take into account the original job_name when creating an unique key per each scrape target ... This should handle the case when the original job_name has been changed in -promscrape.config , while the resulting job label remains the same because it is overriden via relabeling.	2021-12-20 18:38:05 +02:00
Aliaksandr Valialkin	afafeb379a	all: typo fix: unexected -> unexpected	2021-12-20 17:39:52 +02:00
Aliaksandr Valialkin	5a36e241f4	lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase ... This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981	2021-12-20 17:25:11 +02:00
Aliaksandr Valialkin	8a7f08ded3	lib/storage: properly update per-part min_dedup_interval file contents after merge ... Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value This is a follow-up for 4ff647137a	2021-12-17 20:13:24 +02:00
Aliaksandr Valialkin	a3adf24527	lib/promscrape: allow up to 5 redirects when scraping a target by default ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:14 +02:00
Aliaksandr Valialkin	4ff647137a	lib/storage: deduplicate samples more thoroughly ... Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in higher disk space usage.	2021-12-15 15:59:58 +02:00
Aliaksandr Valialkin	92070cbb67	lib/storage: return dedup interval in milliseconds from GetDedupInterval() ... This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:26:38 +02:00
Aliaksandr Valialkin	1d20a19c7d	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() ... This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:49:12 +02:00
Aliaksandr Valialkin	e1a715b0f5	lib/storage: convert alternate regexps into Graphite wildcards inside __graphite__ pseudo-label ... For example, `{__graphite__=~"foo.(bar|baz)"}` is automatically converted to `{__graphite__=~"foo.{bar,baz}"}` before execution. This allows using multi-value Grafana template variables such as `{__graphite__=~"foo.($app)"}`.	2021-12-14 19:51:49 +02:00
Yury Molodov	c1fd93e8a0	vmui: multiple queries (#1916) ... * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * feat: remove save the last query to local storage * fix: handle an error in a table with subqueries * feat: store display type in URL * Revert "feat: store display type in URL" This reverts commit ccc242c69a. * feat: store display type in URL * refactor: move the time setting to a folder * refactor: move the query configurator to a folder * refactor: move the auth settings to a folder * feat: improve styles * feat: add multi query * update package-lock * feat: add display multiple queries * feat: add limits for multiple queries * update dependencies * feat: add history for multiple queries * feat: add line type to legend * feat: change style for switch * feat: change the logic for axes limits for multiple queries * update package-lock.json * update dependencies * feat: add the filter to legend * wip * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-08 16:40:15 +02:00
Aliaksandr Valialkin	45d082bbe2	app/vminsert: add -maxLabelValueLen command-line flag ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908	2021-12-06 11:40:34 +02:00
Aliaksandr Valialkin	da402fbdfa	lib/workingsetcache: fix unaligned 64-bit atomic operation panic on 32-bit architectures ... The panic has been introduced in 7275ebf91a	2021-12-03 01:21:51 +02:00
Aliaksandr Valialkin	06642d97f5	app: allow specifying http and https urls in the following command-line flags ... * -promscrape.config * -relabelConfig * -remoteWrite.relabelConfig * -remoteWrite.urlRelabelConfig	2021-12-03 00:10:02 +02:00
Aliaksandr Valialkin	62b4efb3e7	app/vmauth: follow-up for 13368bed18 ... * Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md * Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file should be read from the given path. For example, in `-promscrape.config` at `vmagent`.	2021-12-02 23:32:05 +02:00
Aliaksandr Valialkin	394a345ae0	lib/httpserver: expose /-/healthy and /-/ready endpoints as Prometheus does ... This improves integration with third-party solutions, which rely on these endpoints. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833	2021-12-02 14:36:58 +02:00
Aliaksandr Valialkin	90c542af12	app: use relative paths instead of absolute paths for the supported http handlers on the main page ... This allows hiding VictoriaMetrics components behind proxies, which serve pages at different path prefixes See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858	2021-12-02 13:52:39 +02:00
Aliaksandr Valialkin	03f5ad3060	lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp	2021-12-02 13:43:49 +02:00
Aliaksandr Valialkin	49a18b8660	lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865	2021-12-02 13:33:26 +02:00
Aliaksandr Valialkin	c0cbf0de2a	app/{vmbackup,vmrestore}: export internal metrics at /metrics http handler	2021-12-02 11:55:58 +02:00
Aliaksandr Valialkin	7275ebf91a	app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches ... The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query: vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9	2021-12-02 10:30:43 +02:00
Aliaksandr Valialkin	2f63dec2e3	lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics ... These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`	2021-12-02 10:30:42 +02:00
Aliaksandr Valialkin	2fb5a6ca78	lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges	2021-12-01 11:02:36 +02:00
Nikolay	06eff5a72c	removes FileSize from backup part key (#1872) ... * removes FileSize from backup part key it should fix download restoration for backups * Update lib/backup/common/part.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-01 11:01:28 +02:00
Aliaksandr Valialkin	d666755159	lib/storage: take into account -storage.minFreeDiskSpaceBytes when performing big merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-11-30 12:56:35 +02:00
guidao	f05cddd2fc	fix #1830 (#1861) ... Co-authored-by: wangfeng <wangfeng@zhihu.com>	2021-11-30 01:12:24 +02:00
Aliaksandr Valialkin	ba927d1c77	lib/protoparser/prometheus: follow-up for 8e338632a3 ... Do not spend CPU time on error message formatting if error logger is disabled	2021-11-30 00:50:11 +02:00
Nikolay	8e338632a3	Changes unmarshallRow logger to noop for getRowsDiff (#1835)	2021-11-30 00:48:13 +02:00
Aliaksandr Valialkin	d44c585ca4	lib/protoparser: do not log connection reset by peer error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections ... This error is expected, so there is no need in spamming the log with this error.	2021-11-29 21:47:56 +02:00
Aliaksandr Valialkin	b688960db0	lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent	2021-11-17 16:41:35 +02:00
Lan	b72eed1f5e	Add flag of S3ForcePathStyle (#1802)	2021-11-17 01:03:03 +02:00
Aliaksandr Valialkin	e5ac9d8e57	all: consistently return application/json content-type without charset=utf-8 ... The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:04:44 +02:00
Aliaksandr Valialkin	fd596945e7	lib/promscrape: improve logging for scrape_config_files parse errors ... Log the actual file path, which led to the parse error. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789	2021-11-08 13:34:12 +02:00
Aliaksandr Valialkin	cbfc7b7c92	app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-05 14:41:16 +02:00
Aliaksandr Valialkin	e73a82f7a5	lib/promauth: do not show empty values in oauth2 config section at /config page	2021-11-05 12:53:39 +02:00
Aliaksandr Valialkin	aa534c2582	lib/promscrape: add -promscrape.maxResponseHeadersSize command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:26:56 +02:00
Aliaksandr Valialkin	d1eb87c831	app/{vmagent,vminsert}: add ability to restrict access to /config page with authKey query arg ... The authKey can be configured via `-configAuthKey` command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-01 16:44:54 +02:00
Aliaksandr Valialkin	bb87949d5c	lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp	2021-10-28 12:47:22 +03:00
Aliaksandr Valialkin	d0e7c0535e	lib/logger: show only explicitly set command-line flags in logs ... This reduces initial verbosity in logs	2021-10-28 11:00:52 +03:00
Aliaksandr Valialkin	74b8af9891	lib/promscrape: add collapse and expand buttons per each group of targets from the same scrape job	2021-10-27 20:03:24 +03:00
Aliaksandr Valialkin	6608705652	app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules ... Previously only the lower part of 64-bit hash was used for calculating the offset. This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash for calculating the offset.	2021-10-27 19:59:16 +03:00
Aliaksandr Valialkin	e3a91b186a	lib/protoparser/prometheus: optimize GetRowsDiff() function ... This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 , since the provided profile shows that the majority of CPU and memory is spent in this function during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.	2021-10-27 18:54:45 +03:00
Aliaksandr Valialkin	95d44157fc	lib/protoparser/prometheus: add a benchmark for GetRowsDiff	2021-10-27 18:53:54 +03:00
Aliaksandr Valialkin	1952ab99aa	all: fix build issues and tests for Apple M1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:06:34 +03:00
Aliaksandr Valialkin	4821adfd95	lib/promscrape: properly show proxy_url option value at /config page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:23:54 +03:00
Aliaksandr Valialkin	7fa15f7f86	lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set ... The response body isn't used if -promscrape.noStaleMarkers is set after the commit 2876137c92 , so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694	2021-10-22 16:44:44 +03:00
Aliaksandr Valialkin	6106d4069d	lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed ... This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets This is a follow-up after 9882cda8b9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728	2021-10-22 13:54:30 +03:00
Aliaksandr Valialkin	2876137c92	lib/promscrape: reduce memory usage if -promscrape.noStaleMarkers command-line flag is passed ... Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled. This should reduce memory usage when the scraped targets return large responses.	2021-10-22 13:10:29 +03:00
Nikolay	a3684fe3de	adds tab as second separator for graphite text protocol (#1733) ... * adds tab as second separator for graphite text protocol * changes indexFunc for indexAny * Update lib/protoparser/graphite/parser_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:23:45 +03:00
Aliaksandr Valialkin	8991c8b589	lib/flagutil: do not expose sensitive info (passwords, keys and urls) at /flags page	2021-10-20 00:51:26 +03:00
Aliaksandr Valialkin	8ad95f0db7	lib/httpserver: expose command-line flags at /flags page ... This should simplify debugging. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-20 00:45:09 +03:00
Aliaksandr Valialkin	676ad70d9f	lib/envflag: use flag.Set for setting the flags from env vars ... This should make visible the set flags at flag.Visit(), which is used later for logging and exporting the `is_set` label for these flags at /metrics page	2021-10-20 00:41:08 +03:00
Aliaksandr Valialkin	53bb58ed2a	lib/storage: log a warning when the -storageDataPath has less than -storage.minFreeDiskSpaceBytes ... This should improve the debuggability of the readonly feature. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727	2021-10-19 23:59:13 +03:00
Aliaksandr Valialkin	3408a05d12	lib/promscrape/discovery/kubernetes: log a warning if role: endpoints discovers more than 1000 targets per a single endpoint ... In this case `role: endpointslice` must be used instead. See the following references: * https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity * https://github.com/kubernetes/kubernetes/pull/99975 * https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398	2021-10-19 13:20:40 +03:00
Nikolay	cbcc622786	changes job source for /target api (#1723) ... use jobNameOriginal instead of relabeled as prometheus does https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707	2021-10-19 08:49:36 +03:00
Aliaksandr Valialkin	c37f285466	lib/promscrape: set honor_timestamps: true by default if this option isnt set explicitly in scrape configs ... This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-10-16 20:49:08 +03:00
Aliaksandr Valialkin	c055bc478c	lib/promscrape: expose promscrape_series_limit_max_series and promscrape_series_limit_current_series metrics per each scrape target with the enabled unique series limiter	2021-10-16 18:47:13 +03:00
Aliaksandr Valialkin	06b0982d6b	lib/promscrape: always initialize http client for stream parsing mode ... Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:18:23 +03:00
Aliaksandr Valialkin	32793adbd9	lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse ... This should reduce memory usage when scraping targets with big response bodies.	2021-10-16 13:00:30 +03:00
Aliaksandr Valialkin	9866dd95c1	lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice ... This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode at the cost of the increased memory usage, since now the potentially big response is kept in the lastScrape byte slice per each scrapeWork. In practice the memory usage increase shouldn't be big, since the response size is usually much smaller than the parsed metrics from this response after the relabeling, which usually adds a big pile of target-specific labels per each metric.	2021-10-15 15:39:23 +03:00
Aliaksandr Valialkin	f6d33596ff	lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name ... This is a follow-up for 31b42b30b6	2021-10-15 12:27:12 +03:00
Aliaksandr Valialkin	bbd34fa15e	lib/promscrape: add -promscrape.minResponseSizeForStreamParse command-line option for automatic switching to stream parsing mode when scraping targets with big responses ... This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes. This is common case in Kubernetes monitoring.	2021-10-14 12:29:35 +03:00
Aliaksandr Valialkin	1a7287c408	lib/promscrape: return error if sample_limit or series_limit options are set when stream parsing mode is enabled	2021-10-14 12:11:23 +03:00
Aliaksandr Valialkin	e3c8304deb	lib/promscrape: add ability to show the original labels for discovered targets at /targets page ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698	2021-10-13 15:59:58 +03:00
Roman Khavronenko	c0a932a55f	lib/promscrape: make errcheck happy (#1703)	2021-10-13 14:57:30 +03:00
Aliaksandr Valialkin	9882cda8b9	lib/promscrape: shard targets among cluster nodes after relabeling is applied ... This guarantees that targets with the same set of labels go to the same vmagent node. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495	2021-10-12 17:06:00 +03:00
Aliaksandr Valialkin	5a58c041c2	app/vmagent: expose -promscrape.config contents at /config page as Prometheus does ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-12 16:25:37 +03:00
Aliaksandr Valialkin	873aac584e	lib/promscrape: use Prometheus format for target labels at /targets page ... This should simplify copy-pasting the labels to/from PromQL / MetricsQL	2021-10-11 12:41:37 +03:00
Aliaksandr Valialkin	001750c239	lib/storage: fix unaligned access on 32-bit architectures. ... The bug has been introduced at a171916ef5	2021-10-08 19:43:03 +03:00
Aliaksandr Valialkin	cf5cbd1c70	app/{vminsert,vmstorage}: follow-up after a171916ef5 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-10-08 14:35:49 +03:00
Nikolay	4290b46e8c	Adds read-only mode for vmstorage node (#1680) ... * adds read-only mode for vmstorage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269 * changes order a bit * moves isFreeDiskLimitReached var to storage struct renames functions to be consistent change protoparser api - with optional storage limit check for given openned storage * renames freeSpaceLimit to ReadOnly	2021-10-08 14:35:48 +03:00
Ziqi Zhao	402c995d6d	fix some typos (#1678) ... Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>	2021-10-06 14:43:10 +03:00
Aliaksandr Valialkin	6ee66fb6b1	lib/promscrape: reduce memory allocations in mergeLabels() after 48e3e6c8df	2021-09-30 16:56:12 +03:00
Aliaksandr Valialkin	463a5bf76e	lib/protoparser: go fmt	2021-09-29 21:19:00 +03:00
Aliaksandr Valialkin	58964d52a5	lib/protoparser/prometheus: compare invalid Prometheus lines in full	2021-09-29 19:41:28 +03:00
Aliaksandr Valialkin	d80d72efec	app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS ... The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls, while leaving support for `gcs://` urls for backwards compatibility.	2021-09-29 12:10:29 +03:00
Nikolay	3d17112a7e	changes auth validation for openstack (#1663) ... * changes auth validation for openstack must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 00:28:49 +03:00
Aliaksandr Valialkin	91b3c601bc	app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API ... See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206	2021-09-29 00:13:08 +03:00
Aliaksandr Valialkin	718eca33ab	lib/storage: properly handle {__name__=~"prefix(suffix1|suffix2)",other_label="..."} queries ... They were broken in the commit 00cbb099b6 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644	2021-09-23 21:48:51 +03:00
Aliaksandr Valialkin	a0313c046b	lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:47:54 +03:00
Aliaksandr Valialkin	9ca1cbced1	lib/httpserver: add -enterprise and/or -cluster suffixes to short_version label of vm_app_version metric ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635	2021-09-21 23:12:42 +03:00
Aliaksandr Valialkin	207c5760ce	lib/promrelabel: fix parsing regex: true in relabeling rules	2021-09-21 23:00:53 +03:00
Nikolay	ad08d9dfc0	changes protoparser apis for accepting reading from io.Reader (#1624) ... adds InsertHandlerForReader apis to vmagent	2021-09-20 14:49:28 +03:00
Nikolay	0e09fdb8b0	makes filters optional for ec2 api requests (#1627) ... filters can be applied only for DescribeInstances requests, like prometheus does. related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2021-09-17 18:00:37 +03:00
Aliaksandr Valialkin	8f685d81c6	lib/storage: follow up after 00cbb099b6	2021-09-14 14:16:25 +03:00
faceair	00cbb099b6	lib/storage: optimize convert multiple values regexp filter to composite tag filter (#1610) ... * lib/storage: optimize convert multiple values regexp filter to composite tag filter * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-09-14 12:47:07 +03:00
Aliaksandr Valialkin	7f0a8d4bdb	docs: consistency renaming: Influx -> InfluxDB	2021-09-13 17:05:16 +03:00
Aliaksandr Valialkin	fb6ed0ce19	lib/promscrape/discovery/docker: support host networking mode ... See https://github.com/prometheus/prometheus/issues/9116	2021-09-13 13:30:16 +03:00
Aliaksandr Valialkin	6295861acd	lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery	2021-09-13 13:03:42 +03:00
Aliaksandr Valialkin	728c4c3841	lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does ... See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:20:44 +03:00
Aliaksandr Valialkin	0b4eb0fa7d	lib/promscrape: make fmt	2021-09-12 13:34:15 +03:00
Aliaksandr Valialkin	48e3e6c8df	lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling ... See https://github.com/prometheus/prometheus/pull/8911	2021-09-12 13:33:41 +03:00
Aliaksandr Valialkin	f3e89754a9	lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric ... Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. The main idea is to perform the calculations on scrape_series_added and series_limit only if the set of metrics exposed by the target has been changed. Scrape targets rarely change the set of exposed metrics, so this optimization should reduce CPU usage in general case.	2021-09-12 12:53:14 +03:00
Aliaksandr Valialkin	cebcb15ba4	lib/storage: verify that the tsidsFound contain the needed tsids in tests added at f4dead529f	2021-09-11 10:57:13 +03:00
Aliaksandr Valialkin	9286107e82	lib/promscrape: send stale markers for disappeared metrics like Prometheus does	2021-09-11 10:51:04 +03:00
Aliaksandr Valialkin	f4dead529f	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:09:21 +03:00
Aliaksandr Valialkin	4aeb8db83f	lib/promscrape: add ability to set series_limit and stream_parse options via relabeling ... This allows managing these options on a per-target basis. Typical use case: to manage these options for pods via Kubernetes annotations.	2021-09-09 18:49:39 +03:00
Aliaksandr Valialkin	468f941f7e	lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric	2021-09-09 17:37:37 +03:00
Aliaksandr Valialkin	086b5d0cf1	lib/promscrape: add scrape_ prefix to job and target labels exported by promscrape_series_limit_rows_dropped_total metric ... This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.	2021-09-09 17:29:21 +03:00
Aliaksandr Valialkin	d6bd956930	lib/promrelabel: add keep_metrics and drop_metrics actions to relabeling rules ... These actions simlify metrics filtering. For example, - action: keep_metrics regex: 'foo|bar|baz' would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted. The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above: - action: keep_metrics regex: - foo - bar - baz	2021-09-09 16:18:21 +03:00
Aliaksandr Valialkin	f77dde837a	lib/promscrape: add the ability to limit the number of unique series per each scrape target ... The number of series per target can be limited with the following options: * Global limit with `-promscrape.maxSeriesPerTarget` command-line option. * Per-target limit with `max_series: N` option in `scrape_config` section. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561	2021-09-01 16:03:59 +03:00
Aliaksandr Valialkin	5c63d69454	lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator ... This is a follow up commit after 31b42b30b6	2021-08-29 12:37:03 +03:00
Aliaksandr Valialkin	db330232ac	lib/protoparser/opentsdb: follow-up after 8ee75ca45a	2021-08-29 11:49:21 +03:00
envzhu	8ee75ca45a	lib/protoparser/opentsdb: accept multiple spaces between fields in a row as a deliminator. (#1575)	2021-08-29 11:38:32 +03:00
Aliaksandr Valialkin	31b42b30b6	lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus ... See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)	2021-08-29 11:23:08 +03:00
Aliaksandr Valialkin	2e001db4de	lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices ... This should fix service discovery for these roles in Kubernetes v1.22 and newer versions. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122 The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205	2021-08-29 11:16:59 +03:00
Aliaksandr Valialkin	10f960fa0c	lib/promscrape: add ability to load scrape configs from multiple files ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559	2021-08-26 08:51:16 +03:00
Aliaksandr Valialkin	c27ee35c5c	lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config	2021-08-25 13:05:49 +03:00
Aliaksandr Valialkin	ffc0ab1774	lib/{mergeset,storage}: improve the detection of the needed free space for background merge ... This should prevent from possible out of disk space crashes during big merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560	2021-08-25 09:35:44 +03:00
Aliaksandr Valialkin	d5622b32e2	lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets ... Store the scraped response body instead of storing the parsed and relabeld metrics. This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics. This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics. This should also reduce CPU usage, since the marshaling of the parsed and relabeld metrics has been substituted by response body copying. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-21 21:17:26 +03:00
Aliaksandr Valialkin	f46a73dcdd	lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape ... This will make timestamps for stale markers more consistent for timestamps for other samples	2021-08-19 14:18:05 +03:00
Aliaksandr Valialkin	c09446a9aa	lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does	2021-08-18 21:59:03 +03:00
Aliaksandr Valialkin	cdc372bb98	app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries ... This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:59:02 +03:00
Aliaksandr Valialkin	226143f31b	lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag ... This option can be useful when vmagent consumes too much additional memory for staleness markers functionality and when staleness markers aren't needed.	2021-08-18 13:43:21 +03:00
Aliaksandr Valialkin	03c959f1df	lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets ... This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509	2021-08-17 00:55:51 +03:00
Aliaksandr Valialkin	a0e18f06eb	lib/promscrape: restore red highlighting for DOWN targets at /targets page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461	2021-08-15 16:03:57 +03:00
Aliaksandr Valialkin	4401464c22	all: add support for Prometheus staleness markers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin	d375d9b878	lib/envflag: add a link to docs for -envflag.enable	2021-08-11 10:29:33 +03:00
Aliaksandr Valialkin	d826352688	app/vmagent: follow-up after fe445f753b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491	2021-08-05 09:52:32 +03:00
Omar Ghader	46e27d60a6	feature: Add multitenant for vmagent (#1505) ... * feature: Add multitenant for vmagent * Minor fix * Fix rcs index out of range * Minor fix * Fix multi Init * Fix multi Init * Fix multi Init * Add default multi * Adjust naming * Add TenantInserted metrics * Add TenantInserted metrics * fix: remove unused metrics for vmagent * fix: remove unused metrics for vmagent Co-authored-by: mghader <marc.ghader@ubisoft.com> Co-authored-by: Sebastian YEPES <syepes@gmail.com>	2021-08-05 09:52:31 +03:00
Aliaksandr Valialkin	50663ba41f	lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 ... See https://github.com/prometheus/prometheus/pull/8978	2021-08-03 16:11:49 +03:00
Aliaksandr Valialkin	3cad8b4564	lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does	2021-08-03 16:11:49 +03:00
Aliaksandr Valialkin	d05cac6c98	li/storage: re-use the per-day inverted index search code for searching in global index ... This allows removing a big pile of outdated code for global index search. This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486	2021-07-30 10:31:37 +03:00
Aliaksandr Valialkin	8ee8660ac4	app/vmselect: follow-up for 626073bca8 ... * Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries * Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes * Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md	2021-07-28 18:00:23 +03:00
Nikolay	9d45b46f4c	adds check for region with custom s3 endpoint (#1465)	2021-07-27 12:35:38 +03:00
Aliaksandr Valialkin	c2deee9911	lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-27 12:04:50 +03:00
Aliaksandr Valialkin	bb31117555	lib/promrelabel: add tests for verifying that regex works as expected in single quotes and double quotes	2021-07-27 10:50:55 +03:00
Aliaksandr Valialkin	8b7917cd81	all: add go:build lines for Go1.17 ... See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin	1318736ad1	lib/promscrape: add missing whitespace at /targets page before up word	2021-07-26 12:22:59 +03:00
Aliaksandr Valialkin	4ba3fd9e6d	lib/workingsetcache: switch from split cache to full cache after the cache size exceeds 95% of split capacity ... Previously the switch occurred when the cache size becomes 100% of its capacity. The cache size could never reach 100% capacity. This could prevent from switching from the split cache to full cache, thus reducing the cache effectiveness.	2021-07-15 16:12:04 +03:00
Aliaksandr Valialkin	d472b03e34	lib/storage: make sure the second call to DeduplicateSamples and deduplicateSamplesDuringMerge doesnt change samples	2021-07-15 12:17:45 +03:00
Aliaksandr Valialkin	682662b2ae	lib/storage: remove cache directory if it contains reset_cache_on_startup file ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447	2021-07-13 17:58:51 +03:00
Aliaksandr Valialkin	2df66dad7b	lib/httpserver: add is_set label to flag metrics ... This label allows determining the set flags with the query `flag{is_set="true"}`	2021-07-13 15:10:13 +03:00
Aliaksandr Valialkin	f9de546139	lib/storage: reset perKeyMisses stats less frequently ... This should reduce CPU usage for queries executed with intervals higher than 30 seconds	2021-07-12 14:33:42 +03:00
Aliaksandr Valialkin	4f80b2f230	lib/storage: properly limit the size of storage/date_metricID cache	2021-07-12 14:25:44 +03:00
Aliaksandr Valialkin	8ca2799478	lib/storage: properly determine when the deduplication is needed in needsDedup ... Previously needsDedup() could return true if the de-duplication wasn't needed for the following case: d < interval / \ | v | v | interval interval Now it properly returns false for this case	2021-07-12 10:53:30 +03:00
Aliaksandr Valialkin	6e0553c92e	lib/mergeset: cache indexBlock items only on the second request ... This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.	2021-07-07 15:23:06 +03:00
Aliaksandr Valialkin	766edbc421	lib/httpserver: print full requestURI in httpserver.Errorf ... This should simplify debugging.	2021-07-07 13:09:40 +03:00
Aliaksandr Valialkin	e843bd7bd7	lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) ... This should reduce the cache size and memory usage for the indexdb/dataBlocks cache	2021-07-07 10:58:51 +03:00
Aliaksandr Valialkin	8b262d4ba7	lib/storage: periodically reset prefetchedMetricIDs cache in order to limit its size under high churn rate	2021-07-07 10:58:51 +03:00
Aliaksandr Valialkin	a7694092b8	Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" ... This reverts commit 7c6d3981bf. Reason for revert: high contention at bucket16Pool on systems with big number of CPU cores. This slows down query processing significantly.	2021-07-06 18:21:35 +03:00
Aliaksandr Valialkin	8aa9bba9bd	lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects ... This should reduce memory usage on systems with big number of CPU cores, since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains a separate pool inmemoryPart objects per each CPU core. Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized by big sizes of inmemoryPart objects.	2021-07-06 16:28:41 +03:00
Aliaksandr Valialkin	7c6d3981bf	lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method ... This reduces the load on memory allocator in Go runtime in production workload.	2021-07-06 15:35:03 +03:00
Aliaksandr Valialkin	78c9174682	lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload ... CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small. This results in the increased load on memory allocation code in Go runtime. Increase the pool capacity in order to reduce the load on Go runtime.	2021-07-06 13:41:34 +03:00
Aliaksandr Valialkin	f71e4d1853	lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds ... This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered (aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.	2021-07-06 12:17:17 +03:00
Aliaksandr Valialkin	f3acf065c9	lib/storage: consistency renaming: tagCache -> tagFiltersCache ... This improves code readability	2021-07-06 11:03:51 +03:00
Aliaksandr Valialkin	0020b9f904	lib/workingsetcache: properly update stats for requests and cache misses ... Previously the stats for cache misses could be improperly counted, because it had inflated cache misses if the entry was missing in the curr cache, but was existing in the prev cache. The same applies to cache requests - they were inflated if the entry was missing in the curr cache.	2021-07-06 10:53:32 +03:00
Aliaksandr Valialkin	4cf47163c1	lib/workingsetcache: fix cache capacity calculations after 4f0003f182	2021-07-05 17:11:57 +03:00
Aliaksandr Valialkin	4f0003f182	lib/workingsetcache: typo fixes after d0c830039d	2021-07-05 15:35:37 +03:00
Aliaksandr Valialkin	d0c830039d	lib/storage: tune cache sizes according to production workload	2021-07-05 15:16:11 +03:00
Aliaksandr Valialkin	8f973e34fb	lib/workingsetcache: properly switch to whole mode ... Previously the switch from `split` to `whole` mode had been performed too early, e.g. when the current cache size became bigger than 1/4 of the allowed cache size. Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size. This change can reduce memory usage for data ingestion path when big number of active time series are ingested.	2021-07-05 15:16:11 +03:00
Aliaksandr Valialkin	43103be011	lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes ... One minute cache timeout result in slower queries in some production workloads where the interval between query execution is in the range 1 minute - 2 minutes.	2021-07-05 15:16:11 +03:00
Aliaksandr Valialkin	54b9e1d3cb	lib/cgroup: set GOGC to 50 by default if it isn't set ... This should reduce memory usage for typical VictoriaMetrics workloads by up to 50%	2021-07-05 15:16:11 +03:00
Aliaksandr Valialkin	9a83e9018d	lib/storage: properly detect free disk space shortage during data merge ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-02 17:40:54 +03:00
Aliaksandr Valialkin	7088f17494	lib/promscrape/discovery/consul: use case-insensitive comparison for service names ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424	2021-07-02 14:49:27 +03:00
Aliaksandr Valialkin	6e406083f2	lib/promauth: cache the client TLS certificate for up to a second ... This should reduce CPU usage when TLS connections are established at a high rate.	2021-07-02 13:21:51 +03:00
Aliaksandr Valialkin	158c50c0ee	lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does ... This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2021-07-01 15:27:47 +03:00
Aliaksandr Valialkin	c25b839078	lib/workingsetcache: reset the cache mode when the cache is reset ... This should reduce memory usage if the working set is reduced after the cache reset.	2021-07-01 11:50:11 +03:00
Nikolay	ae485c2bfd	fixes /targets button style (#1423) ... * fixes /targets button style https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422 * updates boostrap version	2021-07-01 11:48:07 +03:00
Aliaksandr Valialkin	c93cee8de8	lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute ... This should reduce memory usage on a system with high number of active time series and a high churn rate. One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).	2021-06-29 19:57:07 +03:00
Aliaksandr Valialkin	fc12484734	lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs ... This should reduce memory usage for the pool on systems with big number of CPU cores. The sync.Pool maintains per-CPU pools, so the total number of objects in the pool is proportional to the number of available CPU cores. The channel limits the number of pooled objects by its own capacity. This means smaller number of pooled objects on average.	2021-06-29 19:56:59 +03:00
Aliaksandr Valialkin	9ce211a514	lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID	2021-06-29 13:12:28 +03:00
Aliaksandr Valialkin	5506cff76e	lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache ... This should prevent from stats overwriting when the previous indexdb is queried.	2021-06-29 12:40:05 +03:00
Aliaksandr Valialkin	1b0501a09e	lib/promscrape: typo fix in /targets output ... The typo has been introduced in fb72a2133f Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408	2021-06-28 21:26:37 +03:00
Aliaksandr Valialkin	cb5453953f	lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common ... This is a follow up after c85a5b7fcb	2021-06-25 13:20:20 +03:00
Aliaksandr Valialkin	a69045e440	lib/promscrape: consistently sort service discovery routines ... This should simplify further maintenance of the code	2021-06-25 12:10:46 +03:00
Lu Jiajing	c85a5b7fcb	Support Docker ServiceDiscovery (#1402) ... * add docker discovery * add test * add labels test and add scrape work * remove TODO * refactor to merge apiConfig and sdConfig * apply suggestion	2021-06-25 11:42:47 +03:00
Nikolay	434e33da9b	adds missing MustStop call to do and http sd (#1404)	2021-06-25 11:39:18 +03:00
Aliaksandr Valialkin	c22114c6f0	lib/storage: tune tag filters search logic ... Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624 The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats. This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters for further queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-23 13:29:39 +03:00
Aliaksandr Valialkin	e8a5bb92b7	lib/promscrape/discovery/consul: properly pass namespace to Consul watcher ... Follow-up for 58a2989fe7	2021-06-22 17:42:41 +03:00
Aliaksandr Valialkin	ac54f34f9e	lib/promscrape/discovery/http: follow up after e307bbb29a	2021-06-22 13:40:33 +03:00
Aliaksandr Valialkin	755040a171	lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does	2021-06-22 13:34:02 +03:00
Nikolay	e307bbb29a	adds http_sd (#1399) ... * adds http_sd * adds X-Prometheus-Refresh-Interval-Seconds header * Update lib/promscrape/discovery/http/api.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 13:33:37 +03:00
Nikolay	58a2989fe7	adds consul enterprise namespace support (#1400) ... * adds consul enterprise namespace support * Update lib/promscrape/discovery/consul/consul.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 12:49:44 +03:00
Aliaksandr Valialkin	fb72a2133f	lib/promscrape: show jobs with empty scrape targets on /targets page	2021-06-18 10:53:52 +03:00
Nikolay	6c434b260e	fixes DO service discovery labels (#1389) ... adds test for digitalocean sd	2021-06-17 15:12:20 +03:00
Aliaksandr Valialkin	dcbc22552f	lib/storage: fix infinite loop introduced in aa9b56a046	2021-06-17 14:28:10 +03:00
Aliaksandr Valialkin	aa9b56a046	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores ... VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit 56b6b893ce . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 13:52:08 +03:00
Aliaksandr Valialkin	84fb59b0ba	lib/storage: move deletedMetricIDs set from indexDB to Storage ... This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB). This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation. See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 . Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart. This should be OK in most cases.	2021-06-15 15:04:30 +03:00
Aliaksandr Valialkin	e028ad241a	lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error ... This is a follow-up for af90c3c43b	2021-06-14 15:18:49 +03:00
faceair	af90c3c43b	lib/protoparser: stop read when callback error (#1380)	2021-06-14 15:10:58 +03:00
Aliaksandr Valialkin	36d55bff66	lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377	2021-06-14 14:04:00 +03:00
Nikolay	729c4eeb9c	adds digital ocean sd (#1376) ... * adds digital ocean sd config * adds digital ocean sd https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367 * typo fix	2021-06-14 13:15:04 +03:00
Aliaksandr Valialkin	06b8e7d148	lib/promscrape: increase the duration for reading the full response in stream parsing mode ... Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:28:09 +03:00
Aliaksandr Valialkin	48210130ac	lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:25:52 +03:00
Aliaksandr Valialkin	3c4366806c	lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error ... This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:22:04 +03:00
Aliaksandr Valialkin	ed83558646	app/vmauth: properly handle http.ErrAbortHandler panic ... This panic can be raised by the reverseProxy on aborted request to the backend. So handle it (e.g. suppress) at reverseProxy.ServeHTTP call. Do not suppress the panic at lib/httpserver generic HTTP handler, since it may result in an inconsistent state left after the panicking handler. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353	2021-06-11 12:50:25 +03:00
Aliaksandr Valialkin	c4f3fbfa5d	lib/storage: reset cache on disk during series deletion and during indexdb rotation ... This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347	2021-06-11 12:42:28 +03:00
Aliaksandr Valialkin	69b1482bdb	lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard	2021-06-11 10:57:23 +03:00
Aliaksandr Valialkin	044ab46824	lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores	2021-06-11 10:57:23 +03:00
Nikolay	6b29b955c0	disables panic for net/httpAbortHandler (#1355)	2021-06-09 12:08:58 +03:00
Aliaksandr Valialkin	96b691a0ab	lib/storage: properly account the number of loops spent when matching for or suffixes ... This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-08 13:06:12 +03:00
Aliaksandr Valialkin	661d2668f8	lib/promrelabel: add tests for labelsToString() function	2021-06-04 20:42:46 +03:00
Aliaksandr Valialkin	78f83dc5ad	app/{vmagent,vminsert}: follow-up after 2fe045e2a4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343	2021-06-04 20:27:58 +03:00
jelmd	2fe045e2a4	new feature: debug relabeling (#1344) ... * new feature: relabel logging Use scrape_configs[x].relabel_debug = true to log metric names inkl. labels before and after relabeling. After relabeling related metrics get dropped, i.e. not submitted to servers. * vminsert wants relabel logging, too.	2021-06-04 17:50:23 +03:00
Hason Chan	6f19bb23a1	fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350)	2021-06-04 11:47:17 +03:00
Aliaksandr Valialkin	2d8bd41f8a	lib/storage: reduce memory allocations when syncing dateMetricIDCache	2021-06-03 16:20:42 +03:00
Nikolay	ddc8022702	fixes solaris build (#1345)	2021-05-31 09:21:23 +03:00
Aliaksandr Valialkin	a52a20659a	lib/promscrape: fix tests after f0c21b6300	2021-05-28 01:32:50 +03:00
Aliaksandr Valialkin	d088923aef	Revert "lib/mergeset: remove a pool for inmemoryBlock structs" ... This reverts commit 793fe39921. Reason to revert: production testing revealed possible slowdown when registering big number of new time series	2021-05-28 01:09:32 +03:00
Aliaksandr Valialkin	793fe39921	lib/mergeset: remove a pool for inmemoryBlock structs ... The pool for inmemoryBlock struct doesn't give any performance gains in production workloads, while it may result in excess memory usage for inmemoryBlock structs inside the pool during background merge of indexdb.	2021-05-27 21:57:33 +03:00
Aliaksandr Valialkin	60341722d5	docs: document f0c21b6300	2021-05-27 15:03:30 +03:00
faceair	f0c21b6300	lib/promscrape: apply body size & sample limit to stream parse (#1331) ... * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 14:52:44 +03:00
Aliaksandr Valialkin	2233d6ed8a	lib/uint64set: store pointers to bucket16 instead of bucket16 objects in bucket32 ... This speeds up bucket32.addBucketAtPos() when bucket32.buckets contains big number of items, since the copying of bucket16 pointers is much faster than the copying of bucket16 objects. This is a cpu profile for copying bucket16 objects: 10ms 13.43s (flat, cum) 32.01% of Total 10ms 120ms 650: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 651: b.b16his[pos] = hi . 13.31s 652: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . . 653: b16 := &b.buckets[pos] . . 654: *b16 = bucket16{} . . 655: return b16 . . 656:} This is a cpu profile for copying pointers to bucket16: 10ms 1.14s (flat, cum) 2.19% of Total . 100ms 647: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 648: b.b16his[pos] = hi 10ms 700ms 649: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . 330ms 650: b16 := &bucket16{} . . 651: b.buckets[pos] = b16 . . 652: return b16 . . 653:}	2021-05-25 14:20:52 +03:00
Aliaksandr Valialkin	39ef1e7a51	lib/storage: do not stop data ingestion on the first error in Storage.AddRows ... Continue data ingestion for the rest of blocks.	2021-05-24 15:32:47 +03:00
Aliaksandr Valialkin	4b01c9fb2e	lib/storage: limit the number of rows per each block in Storage.AddRows() ... This should reduce memory usage when ingesting big blocks or rows.	2021-05-24 15:24:07 +03:00
Aliaksandr Valialkin	a4ff4b8e65	lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows ... This should reduce memory usage a bit on data ingestion path	2021-05-24 15:22:59 +03:00
Aliaksandr Valialkin	a46551245c	lib/bloomfilter: fix TestLimiterConcurrent	2021-05-24 05:17:36 +03:00
Aliaksandr Valialkin	93d81b486d	lib/fs: do not pass done callback to tryRemoveAll() func ... This improves code readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-24 04:51:57 +03:00
Aliaksandr Valialkin	f54133b200	lib/storage: do not populate MetricID->MetricName cache during data ingestion ... This cache isn't needed during data ingestion, so there is no need in spending RAM on it. This reduces RAM usage on data ingestion path by 30%	2021-05-24 03:02:46 +03:00
Aliaksandr Valialkin	ec79abc382	lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds ... Log these messages if the merge takes more than 30 seconds instead of 10 seconds.	2021-05-23 14:03:21 +03:00
Aliaksandr Valialkin	78dddfb98f	lib/promauth: follow-up after 5b8176c68e	2021-05-22 18:01:11 +03:00
Nikolay	5b8176c68e	basic OAuth2 support for remoteWrite and scrape targets (#1316) ... * adds OAuth2 support for remoteWrite and scrapping * adds tests changes init	2021-05-22 16:20:18 +03:00
Aliaksandr Valialkin	e05dd475f0	lib/fs: concurrently remove up to 1024 blocked NFS directories ... Previously the blocked directories were removed sequentially by a single goroutine. This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second, when big number of LSM parts are created and removed at high rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:57:46 +03:00
Aliaksandr Valialkin	8e2985b53d	lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full ... This should reduce the probability of the panic on a highly loaded VictoriaMetrics accepting millions of samples per second. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:21:00 +03:00
Aliaksandr Valialkin	c54bb73867	all: do not skip SIGHUP signal during service initialization ... This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-21 16:34:06 +03:00
Aliaksandr Valialkin	4c7bb75fa2	Makefile: update golangci-lint from v1.29.0 to v1.40.1	2021-05-20 18:27:10 +03:00
Aliaksandr Valialkin	e394ff6466	app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour ... These numbers are exposed via the following metrics: - vmagent_hourly_series_limit_current_series - vmagent_daily_series_limit_current_series Expose also the limits via the following metrics: - vmagent_hourly_series_limit_max_series - vmagent_daily_series_limit_max_series	2021-05-20 15:28:09 +03:00
Aliaksandr Valialkin	ad73f226ff	app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags	2021-05-20 14:15:19 +03:00
Aliaksandr Valialkin	7e526effaa	app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis	2021-05-20 13:13:40 +03:00
Aliaksandr Valialkin	22585531ad	lib/promscrape/discovery/kubernetes: make golangci-lint happy by removing empty branches	2021-05-20 12:00:29 +03:00
Aliaksandr Valialkin	009e136d88	lib/storage: remove possible data race when logging dropped labels	2021-05-20 02:47:22 +03:00
Aliaksandr Valialkin	eb8093ca6b	lib/promscrape/discovery/kubernetes: reload objects on object parse error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 23:25:48 +03:00
Aliaksandr Valialkin	f4719889da	lib/httpserver: typo fix in -http.shutdownDelay command-line flag description: servier -> server	2021-05-18 16:26:16 +03:00
Aliaksandr Valialkin	0cf1fe19e6	lib/promscrape/discovery/kubernetes: simplify the reload logic for urlWatcher.objectsByKey	2021-05-18 15:40:46 +03:00
Aliaksandr Valialkin	bfb61de606	lib/promscrape/discovery/kubernetes: properly update vm_promscrape_discovery_kubernetes_scrape_works metric ... Previously it wasn't descreased during config update.	2021-05-18 15:33:45 +03:00
Aliaksandr Valialkin	3166994244	lib/promscrape/discovery/kubernetes: log errors and stop service discovery when unexpected updates are received from Kubernetes API server ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 15:10:48 +03:00
Aliaksandr Valialkin	6c944b86d8	docs: dealay -> delay ... Thanks to @jelmd . See 0b7e3510c8 (r50884991)	2021-05-18 01:07:52 +03:00
Aliaksandr Valialkin	ec6a284978	lib/promrelabel: add tests for conditional removal of label on another label match ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1294	2021-05-18 00:23:03 +03:00
Aliaksandr Valialkin	2eed410466	lib/promscrape/discovery/kubernetes: key ScrapeWork objects by urlWatcher instead of namespace ... This makes the code less fragile if urlWatcher would depend on additional to namepsace properties. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-17 23:47:08 +03:00
Aliaksandr Valialkin	733706e6c6	lib/promscrape: reload auth tokens from files every second ... Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:00:08 +03:00
Aliaksandr Valialkin	10a47af631	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport ... By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:12:24 +03:00
Aliaksandr Valialkin	fc3519fa26	lib/promscrape: limit scrape_timeout by scrape_interval like Prometheus does	2021-05-13 16:09:45 +03:00
匠心零度	b4f5be8bd8	fix vagent imbalance problem (#1292) ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-05-13 11:14:51 +03:00
Aliaksandr Valialkin	e6fda03e8f	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:44:14 +03:00
Aliaksandr Valialkin	f6a641de62	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services ... This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:38:50 +03:00
Aliaksandr Valialkin	c0ec541559	lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for b50024812e	2021-05-13 09:26:20 +03:00
Aliaksandr Valialkin	d7be2753c0	lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss	2021-05-13 09:02:33 +03:00
Nikolay	b50024812e	adds cgroupsv2 support (#1283) ... * adds cgroupv2 limits support https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269 * small fix * changes Atoi to ParseUint	2021-05-13 09:02:13 +03:00
Aliaksandr Valialkin	a22a17dc66	lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate ... These functions are non-trivial, while their code has minimal differences. It is better from maintainability PoV to merge these functions into a single function.	2021-05-12 17:56:53 +03:00
Aliaksandr Valialkin	832651c6c2	app/vmselect: follow up after 8a0678678b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168	2021-05-12 17:18:30 +03:00
Nikolay	8a0678678b	Adds tsdb match filters (#1282) ... * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin	cfd6aa28e1	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices scrape targets every 5 seconds, since they may depend on changed service and pod objects ... This should make endpoints and endpointslices scrape targets eventually consistent with the maximum delay of 5 seconds after the related service or pod object changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-12 14:10:34 +03:00
Aliaksandr Valialkin	afec68ad13	lib/httpserver: add new X-Server-Hostname header instead of overwriting already exsiting header ... This makes possible tracking origins of chained requests over multiple hops.	2021-05-11 23:48:59 +03:00
Aliaksandr Valialkin	f2d5c4e2d0	lib/httpserver: return X-Server-Hostname http header in all the responses for better debuggability	2021-05-11 22:03:48 +03:00
Aliaksandr Valialkin	33f7bacb01	lib/storage: properly apply time range when matching an empty filter ... It must match all the time series on the given time range. Previously it was matched to all the time series without the restriction on the given time range.	2021-05-11 01:12:05 +03:00
Aliaksandr Valialkin	75c2c813fc	lib/storage: remove dead code after the commit 3ccf7ea20c	2021-05-08 20:14:11 +03:00
Aliaksandr Valialkin	7aea5f58c4	lib/ingestserver: properly close incoming connections during graceful shutdown	2021-05-08 19:52:58 +03:00
Aliaksandr Valialkin	12d733dd5d	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:52:57 +03:00
Aliaksandr Valialkin	904bbffc7f	lib/promscrape/discovery/kubernetes: start watchers for pods and services before starting watchers for endpoints ... This should eliminate possible race when an update on endpoints depends on pods and/or services, which are missing in the cache yet. This could result in missing targets based on endpoints or endpointslices. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-05 12:23:50 +03:00
Aliaksandr Valialkin	76027093ca	lib/storage: use WARNING instead of INFO level for logging dropped labels	2021-05-03 13:56:43 +03:00
Aliaksandr Valialkin	ce9e163e94	lib/httpserver: stop the process on panics in request handlers ... Panics may leave the process in inconsistent state. That's why it is better to stop the process after the panic instead of recovering from the panic. Unfortunately, the standard net/http.Server recovers panics in request handlers. See https://github.com/golang/go/issues/16542 . That's lib/httpserver must stop the process on itself after the panic.	2021-05-03 11:59:40 +03:00
Nikolay	477369b62f	adds stalePartsRemover (#1261) ... for new created partitions	2021-05-03 11:34:00 +03:00
Aliaksandr Valialkin	9a930720b3	lib/promrelabel: add tests for removing the specified {label="value"} pair	2021-05-03 11:26:53 +03:00
Aliaksandr Valialkin	58d1e6eeea	lib/storage: log dropped labels if the number of labels in a metric exceeds -maxLabelsPerTimeseries command-line flag value ... This should improve debuggability for this case.	2021-05-01 09:27:55 +03:00
Aliaksandr Valialkin	6fa5981e68	app/vmagent: list user-visible endpoints at http://vmagent:8429/ ... While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics	2021-04-30 09:36:43 +03:00
Aliaksandr Valialkin	2ab1266593	lib/promscrape/discovery/kubernetes: remove a mutex at urlWatcher - use groupWatcher mutex for accessing all the urlWatcher children ... This simplifies the code a bit and reduces the probability of improper mutex handling and deadlocks.	2021-04-29 10:14:26 +03:00
Nikolay	4e5a88114a	vmagent kubernetes_sd tests (#1253) ... * first part of tests for kubernetes sd * makes linter happy * added more test cases * adds pub/sub for tests	2021-04-29 10:10:24 +03:00
Aliaksandr Valialkin	87179c6839	lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures ... The panic has been introduced in 56b6b893ce	2021-04-27 16:41:32 +03:00
Aliaksandr Valialkin	56b6b893ce	lib/mergeset: split rows ingestion among multiple shards ... This improves rows ingestion on systems with many CPU cores by reducing lock contention. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244 Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243	2021-04-27 15:36:34 +03:00
Aliaksandr Valialkin	2ec7d8b384	lib/promscrape/discovery/kubernetes: fix a deadlock introduced in eddba29664 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 Thanks to @f41gh7 for providing the initial idea for deadlock fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1248	2021-04-27 14:57:51 +03:00
Aliaksandr Valialkin	f89c1f7f49	lib/storage: typo fix in info message when deleting the part outside the configured retention ... Previously the message was displaying incorrect retention time	2021-04-27 13:32:46 +03:00
Aliaksandr Valialkin	60947fb2d5	lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value	2021-04-27 00:25:52 +03:00
Aliaksandr Valialkin	908e35affd	lib/promscrape: apply scrape_timeout on receiving the first response byte for stream_parse: true scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047	2021-04-23 21:53:35 +03:00
Aliaksandr Valialkin	eddba29664	lib/promscrape/discovery/kubernetes: refresh role: endpoints targets on service object removal as Prometheus does ... This is a follow-up for ae37cfd528 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:26:57 +03:00
Aliaksandr Valialkin	ae37cfd528	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices targets on service object update like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:11:40 +03:00
Aliaksandr Valialkin	bbebdf9ba1	lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142	2021-04-22 13:02:44 +03:00
Aliaksandr Valialkin	6bc52fe41a	all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com	2021-04-20 20:16:17 +03:00
Aliaksandr Valialkin	9b1999a5ff	lib/uint64set: remove memory allocation in getOrCreateSmallPool() ... This partially reverts fb82c4b9fa It has been appeared that the additional memory allocation may result in higher GC pauses. It is better to spend CPU time on copying bigger bucket16 structs instead of increasing query latencies due to higher GC pauses	2021-04-14 12:30:19 +03:00
Aliaksandr Valialkin	251747e253	lib/storage: code clarification: remove caching the found metricName in searchMetricName	2021-04-13 10:22:21 +03:00
Artem Navoiev	77be3e3a82	improve docs for cli flags (#1202) ... * improve docs for cli flags * improve docs for cli flags.2	2021-04-12 12:28:04 +03:00
Aliaksandr Valialkin	3f0bcbe067	lib/promscrape: create a single swosFunc per scrape_config	2021-04-08 09:31:48 +03:00
Aliaksandr Valialkin	5a0938d807	lib/promscrape: do not spend CPU time on constructing scrapeWork key if clustering is disabled	2021-04-07 21:54:22 +03:00
Aliaksandr Valialkin	df32d2836c	lib/storage: properly handle big time ranges passed to /api/v1/labels and /api/v1/label/<labelName>/values ... It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch	2021-04-07 13:33:46 +03:00
Aliaksandr Valialkin	59f9960992	lib/promscrape/discovery: remove superflouos check in registerPendingAPIWatchers ... The check `_, ok := uw.aws[aw]; !ok` isn't needed, since aw cannot exist in uw.aws because of the check inside subscribeAPIWatcher	2021-04-07 13:07:39 +03:00
Aliaksandr Valialkin	3ec6639bbb	lib/promscrape/discovery/kubernetes: register pending apiWatchers in uw.aws	2021-04-06 11:12:13 +03:00
Aliaksandr Valialkin	5f593b0ed3	lib/promscrape/discovery/kubernetes: remove superflouos mustStart and mustStop functions	2021-04-05 22:44:12 +03:00
Lu Jiajing	b59164cf33	fix access to nil *url.URL (#1180) ... * fix access to nil *url.URL Signed-off-by: Megrez Lu <lujiajing1126@gmail.com> * Update lib/promscrape/discovery/kubernetes/api_watcher.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-05 22:25:31 +03:00
Aliaksandr Valialkin	b46194472f	lib/promscrape/discovery/kubernetes: reduce CPU time spent on registering big number of Kubernetes objects shared among big number of scrape jobs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 22:04:30 +03:00
Aliaksandr Valialkin	a51d0ec6ec	lib/promscrape/discovery/kubernetes: load objects missing in local cache from api seriver in getObjectByRole() ... This should fix possible race for `role: endpoints` and `role: endpointslices` service discovery, when the referred `pod` and `service` objects aren't propagated to urlWatcher cache yet. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182#issuecomment-813353359 for details.	2021-04-05 20:31:17 +03:00
Aliaksandr Valialkin	95dbebf512	lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030	2021-04-05 19:26:11 +03:00
Aliaksandr Valialkin	f010d773d6	lib/promscrape/discovery/kubernetes: synchronously load Kubernetes objects on first access ... Remove async registration of apiWatchers, since it breaks discovering `role: endpoints` and `role: endpointslices` targets, which depend on pod and service objects. There is no need in reloading `endpoints` and `endpointslices` targets if the referenced `pod` or `service` objects change, since in this case the corresponding `endpoints` and `endpointslices` objects should also change because they contain ResourceVersion of the referenced `pod` or `service` objects, which is modified on object update. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 14:20:12 +03:00
Aliaksandr Valialkin	2c25b0322b	lib/proxy: typo fix after a5c5b54c22	2021-04-05 13:45:24 +03:00
Aliaksandr Valialkin	a5c5b54c22	lib/proxy: add support for socks5 over tls proxy	2021-04-05 13:00:05 +03:00
Aliaksandr Valialkin	6742839fd6	lib/promscrape: pass X-Prometheus-Scrape-Timeout-Seconds header to scrape targets as Prometheus does	2021-04-05 12:15:24 +03:00
Nikolay	a4c6a3b3e1	adds socks5 support for fasthttp client (#1178) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1177 Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-04 01:29:54 +03:00
Aliaksandr Valialkin	500e625e8c	lib/promscrape: properly send full url in GET request via simple HTTP proxy ... This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 01:20:06 +03:00
Aliaksandr Valialkin	5153410ced	lib/promscrape: support for simple HTTP proxies without CONNECT method support such as https://github.com/prometheus-community/PushProx ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 00:40:40 +03:00
Aliaksandr Valialkin	4c56b1a6dd	lib/promscrape: add tests for authorization config, which has been added in df148f48b7	2021-04-03 22:13:22 +03:00
Aliaksandr Valialkin	43f9842b6f	lib/proxy: log response body on non-200 response code ... This should improve debuggability for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-03 03:03:55 +03:00
Aliaksandr Valialkin	df148f48b7	lib/promscrape: add support for authorization config in -promscrape.config as Prometheus 2.26 does ... See https://github.com/prometheus/prometheus/pull/8512	2021-04-02 21:17:45 +03:00
Aliaksandr Valialkin	7f9c68cdcb	lib/promscrape: add follow_redirect option to scrape_configs section like Prometheus does ... See https://github.com/prometheus/prometheus/pull/8546	2021-04-02 19:56:40 +03:00
Aliaksandr Valialkin	5b08e6fb16	lib/promscrape/discovery/kubernetes: properly track objects with the same names in multiple namespaces ... This is a follow-up for 12e4785fe8 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:45:32 +03:00
Aliaksandr Valialkin	12e4785fe8	lib/promscrape/discovery/kubernetes: properly discover targets in multiple namespaces ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:28:30 +03:00
Nikolay	fdb8995642	Adds aws ECS credentials support (#1175)	2021-04-02 11:56:40 +03:00
Aliaksandr Valialkin	dc9eafcd02	app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage ... This option can be useful when samples for the same time series are ingested with distinct order of labels. For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.	2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin	e1f699bb6c	lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels	2021-03-31 21:24:46 +03:00
Aliaksandr Valialkin	b75c2ce659	lib/uint64set: improve Set.Has() performance scalability on multi-CPU system ... Do not update bucket32.hint on Set.Has() call, since it leads to memory ping-pong between CPU cores multi-CPU system	2021-03-29 12:33:47 +03:00
Aliaksandr Valialkin	2601cc0fb0	lib/storage: do not update b.nextIdx if no samples are removed because of retention	2021-03-29 12:00:21 +03:00
Aliaksandr Valialkin	f39c84b21f	lib/promscrape/discovery/kubernetes: typo fix in error message	2021-03-26 12:46:14 +02:00
Aliaksandr Valialkin	9761ffd161	lib/promscrape/discovery/kubernetes: properly handle too old resource version error message from Kubernetes watch API	2021-03-26 12:28:10 +02:00
Aliaksandr Valialkin	aa81039b42	app/vmselect: log the metric which trigger rollup result cache reset ... This should help finding the source of stale metrics	2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin	6e855d4b82	lib/storage: tune loopsCountPerMetricNameMatch according to production workload	2021-03-25 13:27:47 +02:00
Aliaksandr Valialkin	d4aadba9fa	app/vmagent: add -promscrape.consul.waitTime command-line flag for configuring Consul service discovery wait time ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144	2021-03-23 19:33:25 +02:00
Aliaksandr Valialkin	3a3d2165f9	lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock ... This should speed up Search.NextMetricBlock a bit	2021-03-23 17:56:49 +02:00
Nikolay	29f9ef9b7f	changes consul_service label value (#1143) ... according to prometheus discovery. It should mitigate issue with case sensetive services https://github.com/hashicorp/consul/issues/5707	2021-03-23 15:35:01 +02:00
Aliaksandr Valialkin	3cfb3a3683	lib/storage: respect the deadline passed to Storage.SearchMetricNames	2021-03-22 23:03:17 +02:00
Aliaksandr Valialkin	8e2afdf568	lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache	2021-03-22 22:49:18 +02:00
Aliaksandr Valialkin	910092ca4d	lib/storage: tune loopsCountPerMetricNameMatch	2021-03-22 12:53:17 +02:00
Aliaksandr Valialkin	726f6ad804	lib/storage: small code simplification after 6cee5338b2	2021-03-18 15:21:13 +02:00
Aliaksandr Valialkin	6cee5338b2	lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars ... The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137	2021-03-18 14:53:47 +02:00
Aliaksandr Valialkin	4fb049bcba	lib/fs: reduce the frequency of failed to remove directory ... due to NFS lock log warnings ... Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.	2021-03-18 13:24:46 +02:00
Aliaksandr Valialkin	45dabfac1b	lib/storage: faster move heavy filters to the end of list	2021-03-17 15:12:13 +02:00
Aliaksandr Valialkin	828669e4e1	all: make golint happy	2021-03-17 00:49:28 +02:00
Aliaksandr Valialkin	ccfb0ae2d3	lib/storage: limit loops count in order to reduce max CPU usage during filter search	2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin	576a80b3d9	lib/storage: do not modify filterLoopsCount stats with loopsCount stats ... Such a modification can result in incorrect filter sorting later	2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin	f104f3eb2a	all: make golangci-lint happy after the commit 6378205415	2021-03-17 00:24:40 +02:00
Aliaksandr Valialkin	6378205415	lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics ... This is a follow-up for 18cfc4be7b See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:16:17 +02:00
Nikolay	18cfc4be7b	Adds udp6 support for ingest servers (#1134) ... with flag -enableUDP6 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:03:06 +02:00
Aliaksandr Valialkin	35bb44d317	lib/uint64set: optimize bucket16.addMulti a bit	2021-03-16 21:09:07 +02:00
Aliaksandr Valialkin	fd86a7dc1d	lib/storage: time series search optimization according to production workload profiling ... Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down the function by multiple times when it finds big number of metricIDs (tens of millions).	2021-03-16 20:01:43 +02:00
Aliaksandr Valialkin	e36fbfae5b	lib/storage: further tuning for time series search	2021-03-16 18:46:22 +02:00
Aliaksandr Valialkin	dd7e82c34f	app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate	2021-03-15 22:38:50 +02:00
Aliaksandr Valialkin	37323c57c9	lib/influxutils: return response compatible with InfluxDB 1.8.4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:19:59 +02:00
f41gh7	acf2a82d3c	typo fix	2021-03-15 23:03:52 +03:00
Aliaksandr Valialkin	85a95bf60c	all: various fixes in command-line flag descriptions	2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin	7c37e9aea9	app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 21:37:55 +02:00
Aliaksandr Valialkin	9c77e34ef9	lib/storage: further tuning for time series selector code	2021-03-15 20:31:34 +02:00
Aliaksandr Valialkin	fb82c4b9fa	lib/uint64set: reduce the size of bucket16 by storing smallPool by pointer. ... This reduces CPU time spent on bucket16 copying inside bucket13.addBucketAtPos.	2021-03-15 17:23:31 +02:00
Aliaksandr Valialkin	eb103e1527	lib/uint64set: optimize Set.AddMulti for large sorted sets	2021-03-15 17:10:40 +02:00
Aliaksandr Valialkin	43504ebd14	lib/uint64set: optimize bucket16.add and bucket16.addMulti a bit	2021-03-15 16:58:12 +02:00
Aliaksandr Valialkin	3ccf7ea20c	lib/storage: tune per-day index search	2021-03-15 13:31:55 +02:00
Aliaksandr Valialkin	c14dafce43	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics ... Do not cache too big byte buffers and too big writeRequestCtx objects, since it is cheaper to re-create them instead of wasting RAM for their caching. This reverts 7f6f350ee1	2021-03-15 11:45:39 +02:00
Aliaksandr Valialkin	7f6f350ee1	lib/promscrape: return back the logic for flushing big buffers to storage from the commit 3fd8653b40 ... This should reduce memory usage when vmagent scrapes targets with big number of metrics and `-promscrape.streamParse` isn't enabled	2021-03-14 22:26:00 +02:00
Aliaksandr Valialkin	b88806ecbf	lib/promscrape/discovery/kubernetes: do not start object watcher until initial objects are loaded	2021-03-14 21:55:00 +02:00
Aliaksandr Valialkin	83edbb7cab	lib/promscrape: retry service discovery in a few seconds if it starts returning 0 targets ... This should reduce recovery time from temporary issues during service discovery	2021-03-14 21:53:23 +02:00
Aliaksandr Valialkin	bf15d6a6a2	lib/promscrape: remove duplicate target word in error message	2021-03-14 21:52:02 +02:00
Aliaksandr Valialkin	d409898515	lib/promscrape/discovery/kubernetes: further optimize kubernetes service discovery for the case with many scrape jobs ... Do not re-calculate labels per each scrape job - reuse them instead for scrape jobs with identical Kubernetes role Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-14 21:14:53 +02:00
Aliaksandr Valialkin	7a16e8e3a2	lib/promscrape/discovery: fixes after 133b288681 ... - Removed a deadlock in addAPIWatcher - Do not create unused ScrapeWork objects - Do not spend CPU resources on creating objectByKey map in addAPIWatcher This work is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1125	2021-03-13 15:18:51 +02:00