1647 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after 2c553d5a2f ... - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950) ... * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Roman Khavronenko	f31132b70b	lib/promrelabel: fix expected test result (#2957) ... follow-up after 68c4ec9472 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:48:14 +03:00
Aliaksandr Valialkin	9039f23bd1	lib/promrelabel: do not split regex into multiple lines if it contains groups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:16:15 +03:00
Aliaksandr Valialkin	a17030090b	lib/auth: follow-up after b6a6a659f4	2022-08-07 23:15:25 +03:00
Dmytro Kozlov	a266e3e136	lib/auth: add tests for NewToken function (#2921) ... * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:15:23 +03:00
Aliaksandr Valialkin	fd1ac20760	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:41 +03:00
Aliaksandr Valialkin	77bd4e37cc	lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress ... See 7e65ad3e43 and 7e1111ff14	2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs ... This option was ignored since d289ecded1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	80ecfcf759	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels ... See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin	85b04732ed	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default ... This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin	17290a4598	lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f	2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923) ... * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	db049fed84	lib/mergeset: cleanup after de6dd1cd5a ... Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	ba3ca5d1cd	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:43:24 +03:00
Aliaksandr Valialkin	a1e49606ed	app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin	9f95099cf4	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:13:41 +03:00
Aliaksandr Valialkin	586d267a44	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache ... This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:11:14 +03:00
Aliaksandr Valialkin	962ed46583	lib/mergeset: optimize mergeInmemoryBlocks() function ... Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin	3bbe9054d3	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge ... Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin	547cb1edce	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin	5f2b5bd173	lib/mergeset: add tests and benchmarks for commonPrefixLen function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:25:23 +03:00
Aliaksandr Valialkin	749e825020	lib/pushmetrics: make fmt	2022-07-26 20:41:23 +03:00
Aliaksandr Valialkin	9f1e558c58	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	1b5799f894	lib/promscrape: set up=0 for partially failed scrape in stream parsing mode ... This behaviour aligns with Prometheus behavior	2022-07-22 13:38:25 +03:00
Roman Khavronenko	01755fac38	vmalert: remove dependency on datasource pkg from config (#2905) ... * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 13:38:25 +03:00
Roman Khavronenko	d0abdc2b5b	vmalert: allow configuring custom headers per group (#2901) ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:48:05 +03:00
Aliaksandr Valialkin	f00a6bf837	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin	2d1366353c	lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:42 +03:00
Boris Petersen	61e5f89cfb	fix assume role when running in ECS. (#2876) ... This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:37:33 +03:00
Aliaksandr Valialkin	979444b4ed	all: fix other typos in the same way as 6f4d9b2a48 does	2022-07-18 12:10:41 +03:00
zhenyuxie	14c6212a61	fix inmemoryBlock's Less method (#2881)	2022-07-18 12:00:45 +03:00
Nikolay	c007b129cb	lib/promscrape: adds azure service discovery (#2743) ... * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:45:43 +03:00
guidao	f2d24a660b	add next retention metric (#2863) ... Co-authored-by: wangfeng <wangfeng@zhihu.com>	2022-07-13 12:41:22 +03:00
Dmytro Kozlov	5256af2291	lib/mergeset: fix linter error (#2864)	2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin	7cbcbea49d	lib/mergeset: optimize merge speed a bit ... Use heap.Fix instead of heap.Pop + heap.Push when merging blocks	2022-07-12 12:52:36 +03:00
Aliaksandr Valialkin	eab8ebbe11	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin	5794886662	lib/promscrape: properly set Host header when sending requests via http proxy ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin	95add1e8e4	app/{vmagent,vminsert}: follow-up after d19e46de55 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:32:11 +03:00
Aliaksandr Valialkin	4d03ac90fc	lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin	c4cc45d7f8	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/10759	2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin	f9303e494c	lib/promscrape: fix a test after c66f676f3b	2022-07-06 13:25:17 +03:00
Aliaksandr Valialkin	195dccf678	app/vmselect: add ability to query vmselect from another vmselect	2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin	498c6d6e72	lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target ... See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	b4489028f3	lib/storage: typo fix in MetricName.Unmarshal error	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	1ec4dfd678	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics ... This reduces the number of args to vmselectapi calls	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	2e721f7d16	lib/vmselectapi: rename Server.MustClose to more clear Server.MustStop	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	270e555f47	lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	78eeca6f0d	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	5afa54e845	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() ... This improves the API consistency	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	78f9a8aafd	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index ... Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit 55e7afae3a, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:56:55 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability ... 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin	7fc03a1deb	app/vmagent/remotewrite: add -remoteWrite.header command-line flag for setting additional http headers to send to -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin	4fb0f15322	all: readability improvements for query traces ... - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:19:43 +03:00
ttyv	00956e585d	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786) ... Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	7d5d33fd71	lib/storage: return marshaled metric names from SearchMetricNames ... Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin	15da802f5f	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:52:39 +03:00
Aliaksandr Valialkin	399d4c36ae	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin	64505e924d	app/vmstorage: extract vmselect api server into a separate package - lib/vmselectapi ... This opens doors for implementing vmselect api server at vmselect level, so top-level vmselect could query lower-level vmselect nodes in the same way as it queries vmstorage nodes. This will create the ability to create highly available querying architecture when multiple independent VictoriaMetrics clusters with the same data are located in distinct availability zones. In this case we can use top-level vmselect instead of Promxy for simultaneous querying of all the clusters in all the AZs.	2022-06-27 14:20:41 +03:00
Aliaksandr Valialkin	6386f117c8	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin	926fccbb8d	lib/storage: add querytracer to more contexts ... querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 12:53:49 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them ... This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	71b0dfdefa	lib/promscrape: always send stale markers with the real scrape timestamp ... This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets ... This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	fe2269b999	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package ... This package already has the same name, so there is no need in explicit name	2022-06-21 20:24:28 +03:00
Loki's Wager	ca4730c00f	BugFix part_header.go (#2763) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin	288d13af8d	lib/netutil: parallelize background pings for remote addresses ... This should improve the time needed for determining unavailale remote addresses across big numer of ConnPool's. This is a follow-up for a1629bd3be Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-21 13:32:27 +03:00
Aliaksandr Valialkin	a1629bd3be	lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful ... If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped until the background goroutine determines that the given address can be successfully dialed. This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711 This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756 The main differences are: - The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool. This makes possible re-using this feature everywhere lib/netutil.ConnPool is used. - The check doesn't take into account handshake errors for already established connections. Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.	2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin	45e9732764	docs: follow-up after e4d6b750f6	2022-06-20 17:15:52 +03:00
Nikolay	15662c0f29	lib/httpserver: adds flagsAuthKey command-line flag (#2758) ... * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:15:51 +03:00
Aliaksandr Valialkin	b28c6febf9	app/{vminsert,vmselect}: add -vmstorageDialTimeout command-line flag for tuning the maximum time needed for establishing connections to vmstorage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-20 15:17:34 +03:00
Aliaksandr Valialkin	270ad39359	lib/storage: properly take into account already registered series when -storage.maxHourlySeries or -storage.maxDailySeries limits are enabled ... The commit 5fb45173ae takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:53:41 +03:00
Aliaksandr Valialkin	7a79e7c0ef	lib/storage: create per-day indexes together with global indexes when registering new time series ... Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:32:41 +03:00
Aliaksandr Valialkin	88e1221b35	lib/storage: do not register new series if -storage.maxHourlySeries or -storage.maxDailySeries limits are exceeded ... Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:03:02 +03:00
Aliaksandr Valialkin	c5ac176153	lib/storage: reset metric id caches for the previous and the current hour ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:02:51 +03:00
Aliaksandr Valialkin	450aa0ae5a	lib/promrelabel: support action: graphite relabeling ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:25:49 +03:00
Aliaksandr Valialkin	45fa9d798d	app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb	2022-06-14 18:39:00 +03:00
Aliaksandr Valialkin	fb77843639	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:34:13 +03:00
Aliaksandr Valialkin	3167fbc21d	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:28:21 +03:00
Nikolay	e23af8f05c	lib/httpserver: backport changes from master branch (#2697) ... * lib/httpserver: backport changes from master branch adds basicAuth adds authKey check for /metrics and /debug/pprof requests it should improve security for cluster components * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-14 13:02:44 +03:00
Aliaksandr Valialkin	4af43a4a75	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:28:37 +03:00
Aliaksandr Valialkin	61e03f172b	app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them	2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin	cb39eada77	all: improve query tracing coverage for indexdb search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:04:02 +03:00
Howie	4afd7aa695	feat: rule limit (#2676) ... vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 13:15:33 +03:00
Aliaksandr Valialkin	a9ea3fee38	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:16:12 +03:00
Dmytro Kozlov	f2754c3e90	Cardinality explorer (#2625) ... * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:54:27 +03:00
Roman Khavronenko	2b5e1dee91	vmagent: update SD duration histogram metric if SD is active (#2677) ... The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:53:06 +03:00
Roman Khavronenko	5f33445f66	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:02:55 +03:00
Aliaksandr Valialkin	b6e3c12811	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string ... This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin	68b6ddfb14	all: follow-up after 8edb390e21 ... - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin	3dbb19d624	lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534 ... - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin	dd0d773c13	lib/promauth: follow-up after 006b8c7534 ... - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	6c2fb9d8c4	lib/promscrape: add -promscrape.cluster.name command-line flag ... This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 01:11:23 +03:00
Dmytro Kozlov	ce8aade80e	lib/promscrape: adds service discovery visualization for /targets page(#2675) ... * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-04 01:11:23 +03:00
Nikolay	72e43ef2fe	lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 (#2672)	2022-06-04 01:11:23 +03:00
hadesy	28d4624f60	promscrape/discovery: support kubeconfig (#2533)	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	cc226e6ebe	docs/CHANGELOG.md: follow-up after 2177089f94	2022-06-01 14:57:39 +03:00
Roman Khavronenko	e9ee043879	lib/storage: make indexdb/tagFilters cache size configurable (#2667) ... The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Roman Khavronenko	bca90d7148	promrelabel: add support of lowercase and uppercase relabeling actions (#2665) ... * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Aliaksandr Valialkin	fedfc9e686	lib/storage: stop background merge when storage enters read-only mode ... This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin	afced37c0b	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin	386f6110ec	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum ... In this case there is no need in converting int64 to int	2022-06-01 01:43:25 +03:00
Aliaksandr Valialkin	945e9fa8c4	lib/storage: make fmt	2022-05-31 12:42:48 +03:00
Aliaksandr Valialkin	727cc119b6	lib/storage: do not take into account series from the next day when match[] filter is passed to /api/v1/status/tsdb	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	cd1fa2e4cd	issue-2594: use embedded for static files (#2650) ... embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	6add79143b	removed redundant return (fixed linter) (#2647) ... * removed redundant return * updated lint package version	2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin	f149d56ac2	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag ... This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 23:00:30 +03:00
Aliaksandr Valialkin	38beb9fe04	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag ... This is a follow-up for 0fbf59199a See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:07:14 +03:00
阳明	e4df648ea0	lib/storage: Remove the effect of time zone on next retention period (#2568) (#2574)	2022-05-25 15:10:19 +03:00
Roman Khavronenko	7406665fc3	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) ... * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:17:21 +03:00
Boris Petersen	3a8b4fab97	Add ability to sign requests for all AWS services (#2604) ... This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin	116c0b8f2e	docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag	2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin	d8a276fbe4	lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() ... This should reduce potential spikes in the number of established connections in the following cases: - when the connection establishing procedure becomes temporarily slow - after a temporary spike in the rate of ConnPool.Get() calls See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552	2022-05-11 14:11:06 +03:00
Aliaksandr Valialkin	0d0561ca8c	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future ... Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() ... Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit 67b10896d2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin	af0da45d3e	lib/promscrape: rename promscrape_stale_samples_created_total metric to vm_promscrape_stale_samples_created_total, so its name is consistent with the rest of vm_promscrape_ metrics	2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin	9d40bb7137	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section	2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin	2ce1d09135	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call ... Previously filters wheren't passed to this call after the commit 0e09fdb8b0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin	873f55bac5	lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query ... The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: 0e09fdb8b0 d289ecded1	2022-05-05 10:29:47 +03:00
Dmytro Kozlov	4f40dc9829	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535) ... * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:48 +03:00
Nikolay	7e58cba6cf	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458) ... * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:28:37 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530) ... standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Aliaksandr Valialkin	361b08c30e	lib/storage: leave the last sample per each discrete interval during the deduplicaton ... This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin	190c8b463c	lib/netutil: close connections in ConnPool if they are idle for more than 30 seconds ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-05-02 15:01:52 +03:00
Artem Navoiev	11db05a4ff	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487) ... * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:24:12 +03:00
Aliaksandr Valialkin	a436836402	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:58:08 +03:00
Dima Lazerka	837e440865	Fix targetstatus qtpl paths (#2517) ... Ran `make quicktemplate-gen` from the root directory	2022-04-29 11:18:14 +03:00
Aliaksandr Valialkin	aa82987d70	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing ... This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin	a85ef60b4b	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:24:13 +03:00
Aliaksandr Valialkin	4c3cd96db5	lib/promauth: add support for min_version option at tls_config section in the same way as Prometheus does	2022-04-23 00:24:11 +03:00
Aliaksandr Valialkin	808a2f3b61	lib/promauth: add support for proxy_url option at oauth2 section in the same way as Prometheus does	2022-04-23 00:01:53 +03:00
Aliaksandr Valialkin	4ade8511e2	lib/promauth: add support for tls_config section at oauth2 config in the same way as Prometheus does	2022-04-23 00:01:52 +03:00
Aliaksandr Valialkin	c2b13e6a04	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects ... Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin	a89e31b304	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does ... See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin	cc6eae6992	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems ... Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin	60f74dab56	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated ... This is a follow-up after 26b78ad707	2022-04-22 13:23:37 +03:00
Aliaksandr Valialkin	ed1b394a1a	app/vmstorage: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page ... These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin	fea9d1e6ee	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for 2341bd48d7	2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin	1e0517b9cd	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:41:23 +03:00
Aliaksandr Valialkin	1ae16bf671	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:27 +03:00
Aliaksandr Valialkin	e9f08b1e6a	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects ... There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin	909a3ee0e4	lib/promscrape: follow-up after 91e290a8ff	2022-04-20 16:12:26 +03:00
Nikolay	429848a67d	lib/promscrape: reduce latency for k8s GetLabels (#2454) ... replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:12:25 +03:00
Dmytro Kozlov	9dbfd99777	lib/promscrape: simply update UI (#2479) ... * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 15:38:04 +03:00
Aliaksandr Valialkin	45385a5dc6	lib/promscrape: optimize getScrapeWork() function ... Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:34:18 +03:00
Aliaksandr Valialkin	bfa0b8f710	lib/promscrape: use a hash over target labels as a key for dropped targets' map ... This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:23:54 +03:00
Aliaksandr Valialkin	d0bac8e224	all: typo fix: Kuberntes -> Kubernetes	2022-04-20 10:51:41 +03:00
Dmytro Kozlov	17552dba8b	lib/promscrape: Enable filters for endpoint and labels (#2466) ... * lib/promscrape: Enable filters for endpoint and labels * lib/promscrape: cleanup * lib/promscrape: update template * lib/promscrape: move logic filter logic to backend * lib/promscrape: updated placeholder * lib/promscrape: updated placeholder * lib/promscrape: use two different fields for filters, updated form, added error on parsing queries * lib/promscrape: rename functions * lib/promscrape: removed unused values * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-19 18:27:44 +03:00