1794 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	3019ec3da6	lib/querytracer: fix tests after 49ebc48809	2022-12-08 17:21:38 -08:00
Aliaksandr Valialkin	56b8980915	lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin	49ebc48809	lib/querytracer: put the version of VictoriaMetrics in the first message of query trace ... This should simplify further debugging, since the first thing to start the debugging by query trace is to know the version of VictoriaMetrics, which produced this trace.	2022-12-07 09:46:39 -08:00
Pedro Gonçalves	1e0666abb4	Datadog - Add device as a tag if it's present as a field in the series object (#3431) ... * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:06:03 -08:00
Aliaksandr Valialkin	d99d222f0a	lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown	2022-12-05 21:30:48 -08:00
Aliaksandr Valialkin	8189770c50	all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk ... The main purpose of this command-line flag is to increase the lifetime of low-end flash storage with the limited number of write operations it can perform. Such flash storage is usually installed on Raspberry PI or similar appliances. For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data. The in-memory data is searchable in the same way as the data stored on disk. VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal. The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-05 15:16:14 -08:00
Aliaksandr Valialkin	544ea89f91	lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function	2022-12-04 00:01:04 -08:00
Aliaksandr Valialkin	33dda2809b	lib/mergeset: panic when too long item is passed to Table.AddItems()	2022-12-03 23:32:16 -08:00
Aliaksandr Valialkin	932c1f90ae	lib/storage: remove duplicate logging for filepath on errors	2022-12-03 23:15:22 -08:00
Aliaksandr Valialkin	044a304adb	lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args	2022-12-03 23:10:16 -08:00
Aliaksandr Valialkin	cb44976716	lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers ... This simplifies the code	2022-12-03 23:03:08 -08:00
Aliaksandr Valialkin	28e6d9e1ff	lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation	2022-12-03 23:02:10 -08:00
Aliaksandr Valialkin	343c69fc15	lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart ... This allows packing in-memory blocks with different compression levels depending on its contents. This may save memory usage.	2022-12-03 22:46:48 -08:00
Aliaksandr Valialkin	6d87462f4b	lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part ... Previously only data was compressed with the given compressLevel	2022-12-03 22:34:54 -08:00
Aliaksandr Valialkin	f3e3a3daeb	lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part ... This results in more correct reporting of memory usage for in-memory parts	2022-12-03 22:30:36 -08:00
Aliaksandr Valialkin	c4150995ad	lib/mergeset: reduce the time needed for the slowest tests	2022-12-03 22:26:33 -08:00
Aliaksandr Valialkin	45299efe22	lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}	2022-12-03 22:17:46 -08:00
Aliaksandr Valialkin	5ca58cc4fb	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:12 -08:00
Aliaksandr Valialkin	152ac564ab	lib/storage: remove logging redundant path values in a single error message	2022-12-03 22:13:13 -08:00
Aliaksandr Valialkin	93764746c2	lib/filestream: remove logging redundant path values in a single error message	2022-12-03 22:01:51 -08:00
Aliaksandr Valialkin	4f28513b1a	lib/fs: remove logging redundant path values in a single error message	2022-12-03 22:00:20 -08:00
Aliaksandr Valialkin	7c3c08d102	lib/backup: remove logging duplicate path values in a single error message	2022-12-03 21:55:06 -08:00
Aliaksandr Valialkin	14660d4df5	all: typo fix: the the -> the	2022-12-03 21:53:01 -08:00
Aliaksandr Valialkin	ddc3d6b5c3	lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 ... Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message appears in the log and then upgrade to newer release.	2022-12-03 21:17:31 -08:00
Aliaksandr Valialkin	05c65bd83f	lib/storage: speed up search for data block for the given tsids ... Use binary search instead of linear scan for looking up the needed data block inside index block. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-03 20:58:32 -08:00
Aliaksandr Valialkin	299285b147	lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC	2022-12-02 18:52:37 -08:00
Aliaksandr Valialkin	e9636b4c69	lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts ... Move the common code into releasePartsToMerge() method and consistently use it throughout the code.	2022-12-02 18:52:37 -08:00
Aliaksandr Valialkin	f325410c26	lib/promscrape: optimize service discovery speed ... - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin	295c84df66	lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels	2022-11-29 20:27:48 -08:00
Aliaksandr Valialkin	654e94f420	lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:09 -08:00
匠心零度	fa0ce10275	lib/storage: remove extra error check (#3396)	2022-11-28 16:43:31 -08:00
Aliaksandr Valialkin	58d459e8a8	app/{vminsert,vmagent}: follow-up after 53a63c6c4c ... Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:48:14 -08:00
Roman Khavronenko	03d88bc066	vmagent: expose metrics for tracking config state (#3375) ... Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:38:43 +02:00
Aliaksandr Valialkin	95f0266558	lib/promscrape/discovery/gce: do not pass filter arg when discovering zones ... The filter arg isn't supported by zones API in GCE. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202	2022-11-21 22:32:05 +02:00
Aliaksandr Valialkin	353396aa23	lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration ... While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes. This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-17 19:59:13 +02:00
Aliaksandr Valialkin	5955d23232	lib/promscrape: add a benchmark for internLabelStrings()	2022-11-16 23:02:49 +02:00
Aliaksandr Valialkin	a75137c1c2	lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() ... The issue has been introduced in 58b40f514c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 21:23:35 +02:00
Aliaksandr Valialkin	c3362e3db4	lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio ... Reduce the default value of this flag from 1% to 0.2% after 71335e6024 This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:39:39 +02:00
Aliaksandr Valialkin	4106f197f2	lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf ... Previously indexBlock.bhs pointed to the buffer, which could be changed over time. This could result in incorrect time series search over time. This is a follow-up for 58b40f514c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:09:23 +02:00
Aliaksandr Valialkin	58b40f514c	lib/mergeset: remove string allocation and copying when unmarshaling blockHeader ... This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-15 16:30:54 +02:00
Aliaksandr Valialkin	71335e6024	lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1% ... It has been appeared that some production workloads could suffer for some time after every reset of the previous cache when it gets less than 5% of requests after the needed item isn't found in the current cache. This could result in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM usage needed for reading, unpacking and caching the missed data from disk. This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%. This should reduce the possible negative impact after each cache reset by at least 5x, while reducing the total memory used by caches. This is a follow-up for d906d8573e	2022-11-10 13:31:54 +02:00
Aliaksandr Valialkin	86bce7f5f9	lib/promscrape: add more cases to TestAddRowToTimeseries ... This is a follow-up for 16fdd2af8a	2022-11-09 16:13:56 +02:00
Jeremy PLANCKEEL	16fdd2af8a	test(golang): add test to function addRowToTimeseries (#3282) ... Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>	2022-11-09 15:41:26 +02:00
Aliaksandr Valialkin	b8839df32c	lib/protoparser/opentsdb: follow-up after 04b0e4e7bf ... - Simplify the parser code to be less error prone - Document the change - Add a test for OpenTSDB put line with trailing whitespace without tags Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290	2022-11-09 15:35:05 +02:00
Roman Khavronenko	04b0e4e7bf	protoparser/opentsdb: allow lines without tags (#3303) ... According to http://opentsdb.net/docs/build/html/api_telnet/put.html "At least one tag pair must be present". However, in VictoriaMetrics datamodel tags aren't required. This could be confusing for users. Allowing accept lines without tags seems to do no harm. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 15:32:47 +02:00
Aliaksandr Valialkin	7fa5d043f5	lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/11482	2022-11-07 15:25:53 +02:00
Aliaksandr Valialkin	daa70e6560	lib/storage: follow-up for 790768f20b ... - Document the bugfix at docs/CHANGELOG.md - Simplify the bugfix a bit	2022-11-07 14:04:08 +02:00
Aliaksandr Valialkin	f9dc3da9e2	lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8	2022-11-07 13:58:27 +02:00
Aliaksandr Valialkin	116811d761	lib/envtemplate: allow non-env var names inside "%{ ... }"	2022-11-07 13:58:27 +02:00
Aliaksandr Valialkin	dd88c628aa	lib/storage: remove unused isFull field from hourMetricIDs struct	2022-11-07 13:58:26 +02:00
Łukasz Marszał	790768f20b	Fix issue-3309 - currHourMetricIDs shouldn't contain metrics from prev hour (#3320) ... * fix issue-3309 currHourMetricIDs shouldn't contain metrics from prev hour * Update storage.go	2022-11-07 13:55:37 +02:00
Aliaksandr Valialkin	869e0f9f85	lib/promrelabel: go fmt after 5cec9706dc	2022-10-29 05:17:10 +03:00
Aliaksandr Valialkin	5cec9706dc	lib/promrelabel: add a test from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251	2022-10-29 04:33:38 +03:00
Aliaksandr Valialkin	320ae1c60a	lib/envflag: small refactoring after 518c340ae3 and 02096e06d0	2022-10-29 02:28:58 +03:00
Aliaksandr Valialkin	76e8888272	lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set. ... The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function. The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_` prefix to labels inside `labels` slice become invisible in the returned `dst` labels. While at it, properly handle some corner cases: - Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix. - Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278 Thanks to @jplanckeel for the initial attempt to fix this issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281	2022-10-28 22:14:26 +03:00
Aliaksandr Valialkin	454baf84d6	lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page	2022-10-28 22:14:25 +03:00
Aliaksandr Valialkin	518c340ae3	lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax ... This is a follow-up for 02096e06d0	2022-10-26 14:49:33 +03:00
Aliaksandr Valialkin	02096e06d0	lib/envflag: allow referring environment variables in command-line flags	2022-10-26 01:52:05 +03:00
Aliaksandr Valialkin	c4265322f4	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin	d9bbf24183	app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop ... Previously netstorage.MustStop() call didn't free up all the resources, so the subsequent call to nestorage.Init() would panic. This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.	2022-10-25 17:47:17 +03:00
Aliaksandr Valialkin	8e998aa1a1	lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series) ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289	2022-10-24 16:40:20 +03:00
Aliaksandr Valialkin	dba218a8ce	lib/storage: skip blocks outside the configured retention during search ... Blocks outside the configured retention are eventually deleted during background merge. But such blocks may reside in the storage for long time until background merge. Previously VictoriaMetrics could spend additional CPU time on processing such blocks during search queries. Now these blocks are skipped.	2022-10-24 02:52:44 +03:00
Aliaksandr Valialkin	e2f0b76ebf	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg ... This makes code easier to read. This is a follow-up after d2d30581a0	2022-10-24 01:31:04 +03:00
Aliaksandr Valialkin	89a1108b1a	lib/storage: small code cleanups	2022-10-24 01:17:47 +03:00
Aliaksandr Valialkin	05512fdd74	lib/storage: re-use newTestStorage() instead of manually initializing Storage mock ... This is a follow-up for d2d30581a0	2022-10-23 16:24:00 +03:00
Aliaksandr Valialkin	d2d30581a0	lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback ... This improves code readability a bit.	2022-10-23 16:10:04 +03:00
Aliaksandr Valialkin	54f35c175c	lib/storage: small refactoring: move retentionDeadline to blockStreamMerger ... This allows defining per-block retention in the future by updating the getRetentionDeadline function	2022-10-23 16:10:02 +03:00
Aliaksandr Valialkin	187e294a53	lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop	2022-10-23 14:08:57 +03:00
Aliaksandr Valialkin	d0a9ca1bc2	lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms	2022-10-23 12:48:00 +03:00
Aliaksandr Valialkin	5e4dfe50c6	lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function ... The searchTSIDs function was searching for metricIDs matching the the given tag filters and then was locating the corresponding TSID entries for the found metricIDs. The TSID entries aren't needed when searching for time series names (aka MetricName), so this commit removes the uneeded TSID search from the implementation of /api/v1/series API. This improves perfromance of /api/v1/series calls. This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls, since now these calls cache small metricIDs instead of big TSID entries in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs) without the need to compress the saved entries in order to save cache space. This commit also removes concurrency limiter during searching for matching time series, which was introduced in 8f16388428, since the concurrency for all the read queries is already limited with -search.maxConcurrentRequests command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2022-10-23 12:23:47 +03:00
Aliaksandr Valialkin	4128ad71e2	lib/storage: move common code to newRawRowsBlock() function	2022-10-21 14:46:55 +03:00
Aliaksandr Valialkin	b5674164c6	lib/storage: simplify code a bit after 3f5959c053	2022-10-21 14:39:27 +03:00
Aliaksandr Valialkin	fd7c86ae25	lib/{mergeset,storage}: simplify the code a bit after ae55ad8749	2022-10-21 14:33:03 +03:00
Aliaksandr Valialkin	99d67ac8ad	lib/storage: validate timestamps in the block only if they use encoding, which needs validation ... This reduces CPU usage when there is no sense in validating timestamps. This is a follow-up for 5fa9525498 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011	2022-10-21 00:52:32 +03:00
Aliaksandr Valialkin	3f5959c053	lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate ... This should improve background merge rate under high load a bit	2022-10-20 23:28:24 +03:00
Aliaksandr Valialkin	891ff6af2a	lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes ... This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes. This This change shouldn't increase memory usage for caches, since the prev cache cleaner should free up memory by deleting unused prev cache as soon as possible. See 08ca45d238 for details on prev cache cleaner.	2022-10-20 21:48:25 +03:00
Aliaksandr Valialkin	08ca45d238	lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine ... This makes the code more clear after d906d8573e	2022-10-20 21:45:29 +03:00
Aliaksandr Valialkin	4cd173bbaa	lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal ... Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals if they hang for some reason in graceful shutdown procedure.	2022-10-20 21:40:20 +03:00
Aliaksandr Valialkin	150e99d403	lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms ... The panic has been introduced in 68f3a02589 While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs This should improve scalability on systems with many CPU cores	2022-10-20 16:25:43 +03:00
Aliaksandr Valialkin	d906d8573e	lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache ... This means that the majority of requests are successfully served from the current cache, so the previous cache can be reset in order to free up memory.	2022-10-20 10:47:58 +03:00
Aliaksandr Valialkin	817aeafd69	lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses ... This should improve cache scalability on systems with many CPU cores.	2022-10-20 09:12:17 +03:00
Aliaksandr Valialkin	9c02c39487	lib/workingsetcache: randomize interval for swapping curr and prev caches ... This should make CPU usage smoother over time, since different caches will be swapped at different times.	2022-10-20 08:42:43 +03:00
Nikolay	1059c4d84a	lib/promscrape/discovery/kubernetes: correctly wrap error (#3250) ... * lib/promscrape/discovery/kubernetes: correctly wrap error follow-up after 1304824201 * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-18 20:37:42 +03:00
Aliaksandr Valialkin	069401a304	all: log error when environment variables referred from -promscrape.config are missing ... This should prevent from using incorrect config files	2022-10-18 10:47:16 +03:00
Aliaksandr Valialkin	fb50730ba7	lib/storage: double the number of rawRows shards on multi-core systems ... This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage	2022-10-17 18:19:51 +03:00
Aliaksandr Valialkin	ae55ad8749	lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list	2022-10-17 18:01:26 +03:00
Aliaksandr Valialkin	b6e8c1403a	lib/promrelabel: add relabeling tests when the source label is missing	2022-10-17 14:47:52 +03:00
Aliaksandr Valialkin	2e3be68617	lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map ... This prevents from possible corruption of the internal cache map when the underlying byte slice used by the string key is modified. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227	2022-10-14 09:51:19 +03:00
Nikolay	b856581ad3	lib/backup: set s3 default region to us-west-2 (#3224) ... * lib/backup: set s3 default region to us-west-2 it should fix an error with region detection for bucket, if AWS_REGION env var is not set * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-13 10:30:07 +03:00
Aliaksandr Valialkin	185cff307b	lib/mergeset: mention in the error message the path to the part, which triggered the error ... This should improve debuggability	2022-10-12 09:54:21 +03:00
Aliaksandr Valialkin	50f5eae0e0	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places ... Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:51:16 +03:00
Aliaksandr Valialkin	5269b1ad77	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis ... Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:36:14 +03:00
Aliaksandr Valialkin	f9df0cae16	lib/promscrape: allow specifying full target url in __address__ label ... Previously the `__address__` label could contain only `host:port` part of the target url, while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__` labels. Now it is possible to set the full url in `__address__` label. This makes valid the following scrape config, which is frequently used by novice users: scrape_configs: - job_name: foo static_configs: - targets: - http://host1/metrics1 - https://host2/metrics2	2022-10-07 22:43:04 +03:00
Aliaksandr Valialkin	711698b858	lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511	2022-10-07 01:02:06 +03:00
Zakhar Bessarab	176f10f5b2	app/vmbackup: fix compatibility with latest azure sdk (#461)	2022-10-07 01:02:03 +03:00
Aliaksandr Valialkin	d9282027e6	app: follow-up after ec04fcac93 ... * Optimize fast path for /api/v1/import when importing numeric values * Move the docs about the change from features to bugfixes at docs/CHANGELOG.md * Update tests at lib/protoparser/vmimport Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161	2022-10-06 14:52:02 +03:00
Dmytro Kozlov	ec04fcac93	Properly parse json when export import metric (#3180) ... * app/vmselect: properly work when export import json from `api/v1/{export, import}` API * app/vmselect: update convert function * app/vmselect: export null if `math.IsNaN(v)` * app/vmselect: get float from json * lib/protoparser: add test * docs: add change log * lib/protoparser: make export import api compatible	2022-10-06 13:54:20 +03:00
Zakhar Bessarab	97239e05ce	lib/backup/s3remote: fix error checking for alternative S3 providers (#3191)	2022-10-06 13:36:40 +03:00
Aliaksandr Valialkin	1e93ad84e3	lib/backup/azremote: remove unused methods after the 262ce77e2d	2022-10-06 13:08:58 +03:00
Zakhar Bessarab	262ce77e2d	lib/backup: add support of Azure Blob Storage (#460) ... * lib/backup: add support of Azure Blob Storage * lib/backup: add enterprise support of Azure Blob Storage	2022-10-06 00:32:46 +03:00
Aliaksandr Valialkin	0dc93cca7f	app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data ... This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071 Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970	2022-10-01 18:40:59 +03:00
Aliaksandr Valialkin	c1fa9828b3	lib/flagutil: rename Array to ArrayString ... This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:26:36 +03:00
Zakhar Bessarab	87c77727e4	vmbackup: update AWS SDK to v2 (#3174) ... * lib/backup/s3remote: update AWS SDK to v2 * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * lib/backup/s3remote: refactor error handling Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-01 17:12:07 +03:00
Aliaksandr Valialkin	725dfb0ed6	lib/httpserver: use 302 redirects instead of 301 redirects ... Incorrect 301 redirects can be cached by user agents such as web browsers. This can complicate recovery procedure after the incorrect redirect is fixed, e.g. web browser cache must be reset. The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752	2022-10-01 16:53:35 +03:00
Aliaksandr Valialkin	4998402004	lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics ... This aligns with Prometheus behaviour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137	2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin	3a98ef2f5f	lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests	2022-10-01 16:05:51 +03:00
Aliaksandr Valialkin	f86070169d	lib/promscrape/discovery/azure: remove unneeded conversion to string	2022-10-01 16:04:37 +03:00
Aliaksandr Valialkin	db16759c68	lib/storage: optimize matching speed for non-trivial regexp filters ... Wrap re.Match into bytesutil.FastStringMatcher. This increases performance for `{foo=~"complex_regex_here"}` filters by up to 4x.	2022-10-01 12:06:06 +03:00
Aliaksandr Valialkin	e8a64f6e7a	lib/promrelabel: remove redundant memory allocations by using interned strings	2022-10-01 11:50:21 +03:00
Aliaksandr Valialkin	73dc17ef64	lib/promrelabel: add a benchmark for realistic Kubernetes relabeling ... The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes This benchmark has been copied from d521933053/model/relabel/relabel_test.go (L505) See also https://github.com/prometheus/prometheus/pull/11147	2022-10-01 10:38:22 +03:00
Aliaksandr Valialkin	c54e14cdec	lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does ... See https://github.com/prometheus/prometheus/pull/11326	2022-09-30 20:48:32 +03:00
Nikolay	33f40f4a5f	app/vminsert: allows parsing tenant id from labels (#3009) ... * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 18:35:53 +03:00
Aliaksandr Valialkin	171dd14aa3	lib/promrelabel: go fmt	2022-09-30 12:28:55 +03:00
Aliaksandr Valialkin	a18d6d5ccc	lib/promrelabel: optimize action: replace for non-trivial regex values ... Cache `action: replace` results for non-trivial regexs and return them next time instead of performing CPU-intensive regex replacement. Optimize also `action: labelmap_all` and `action: replace_all` in the same way.	2022-09-30 12:25:05 +03:00
Aliaksandr Valialkin	146021a076	lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481	2022-09-30 10:49:18 +03:00
Aliaksandr Valialkin	899d2c40fb	lib/promrelabel: optimize action: labelmap for non-trivial regexs	2022-09-30 10:43:31 +03:00
Aliaksandr Valialkin	17289ff481	lib/regexutil: cache MatchString results for unoptimized regexps ... This increases relabeling performance by 3x for unoptimized regexs	2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin	fda60b3d4d	lib/promrelabel: properly parse regex with escaped $ at the end ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131 Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179	2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin	593da3603e	lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil ... lib/bytesutil is more appropriate place for InternString() function	2022-09-30 07:44:35 +03:00
Nikolay	f61b8cec69	lib/awsapi: fixes sign encoding (#3183) ... * lib/awsapi: fixes sign encoding previously white spaces at filter were incorrectly encoded encoding tip was copied from aws signing lib For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171 * Update lib/awsapi/sign.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 07:43:44 +03:00
Aliaksandr Valialkin	6a32a64073	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin	92b3622253	lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings ... This is a follow-up for 7c2474dac7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105	2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin	ef435f8cc4	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names ... Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for fcffdba9dc Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin	6411bbcce7	lib/netutil/tls.go: consistently use tlsMinVersion name across source code ... This should simplify further code maintenance and refactoring This is a follow-up after 6ab1cede62	2022-09-26 17:58:01 +03:00
Dmytro Kozlov	6ab1cede62	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109) ... * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:35:45 +03:00
Roman Khavronenko	e96ccf3f71	lib/mergeset: follow-up after a0e7432e42 (#3145) ... * lib/mergeset: follow-up after a0e7432e42 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:39:56 +03:00
Zakhar Bessarab	f022296d96	vmbackup: configure retries for GCS remote FS (#3156)	2022-09-26 16:28:20 +03:00
Aliaksandr Valialkin	41f8c2987d	lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102 See the specification https://graphite.readthedocs.io/en/latest/tags.html	2022-09-26 15:17:25 +03:00
Aliaksandr Valialkin	7c2474dac7	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does ... This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:57:23 +03:00
匠心零度	3d5509a720	lib/querytracer: fix comment (#3135)	2022-09-22 19:19:48 +03:00
Aliaksandr Valialkin	56ce7ce85b	lib/promscrape: typo fix after 74c00a8762	2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin	74c00a8762	lib/promscrape: read response body into memory in stream parsing mode before parsing it ... This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin	ccad651a61	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error ... This should improve debugging issues with Kubernetes API server	2022-09-13 19:36:55 +03:00
Aliaksandr Valialkin	ce2c07c5a7	lib/mergeset: atomically remove part dirs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	042a532f70	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	68e32b0764	lib/storage: atomically remove parts inside partitions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	340ada871d	lib/storage: atomically remove partitions, which went outside the configured retention ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin	978dcb4574	lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there ... Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5f28ca1f42	lib/storage: atomically remove snapshot directories ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5fa9525498	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block ... This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:08:09 +03:00
Bryce Lampe	74f8e12e87	Support "HTTP" and "HTTPS" schemes (#3019) ... * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin	30b8d91727	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache ... This is a follow-up for c06e7a142c	2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin	c06e7a142c	lib/promscrape: optimize discoveryutils.SanitizeLabelName() ... Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin	a2cd79576f	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix ... This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:30 +03:00
Aliaksandr Valialkin	f49c9bb700	lib/promrelabel: optimize common regex mismatch cases for action: replace and action: labelmap	2022-08-26 15:45:31 +03:00
Aliaksandr Valialkin	4c6916f32a	lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop,labelkeep,drop and keep ... This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:23:45 +03:00
Aliaksandr Valialkin	7afe8450fc	lib/promrelabel: optimize matching for commonly used regex patterns in if option ... The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo.*" and "foo.+" - substring match, e.g. ".*foo.*" and ".+foo.+" - alternate values match, e.g. "foo|bar|baz"	2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin	0ad3bbadd3	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
Aliaksandr Valialkin	b373661988	lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep} with anchored regex prefix ... The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:23:55 +03:00
Aliaksandr Valialkin	0d4ea03a73	lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values ... For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo|bar|baz"	2022-08-24 17:54:29 +03:00