1720 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	a13f16e48a	lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 17:04:12 -08:00
Aliaksandr Valialkin	255c04bc20	lib/querytracer: put the version of VictoriaMetrics in the first message of query trace ... This should simplify further debugging, since the first thing to start the debugging by query trace is to know the version of VictoriaMetrics, which produced this trace.	2022-12-07 09:49:51 -08:00
Pedro Gonçalves	84cfe4fcaf	Datadog - Add device as a tag if it's present as a field in the series object (#3431) ... * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin	0a9992a9c6	lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown	2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin	7d5c64eb7a	all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk ... The main purpose of this command-line flag is to increase the lifetime of low-end flash storage with the limited number of write operations it can perform. Such flash storage is usually installed on Raspberry PI or similar appliances. For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data. The in-memory data is searchable in the same way as the data stored on disk. VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal. The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin	9ac1174493	lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function	2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin	a13d21513e	lib/mergeset: panic when too long item is passed to Table.AddItems()	2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin	dccd70ce10	lib/storage: remove duplicate logging for filepath on errors	2022-12-03 23:15:28 -08:00
Aliaksandr Valialkin	813e8402f6	lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args	2022-12-03 23:10:26 -08:00
Aliaksandr Valialkin	bb93494eac	lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers ... This simplifies the code	2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin	106332cd9f	lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation	2022-12-03 23:03:30 -08:00
Aliaksandr Valialkin	ea55c16422	lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart ... This allows packing in-memory blocks with different compression levels depending on its contents. This may save memory usage.	2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin	eca7f32151	lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part ... Previously only data was compressed with the given compressLevel	2022-12-03 22:35:16 -08:00
Aliaksandr Valialkin	7ffa66d249	lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part ... This results in more correct reporting of memory usage for in-memory parts	2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin	886ce94739	lib/mergeset: reduce the time needed for the slowest tests	2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin	10a17bfa16	lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}	2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin	233301a549	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin	fd9d0a550b	lib/storage: remove logging redundant path values in a single error message	2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin	82f64072d2	lib/filestream: remove logging redundant path values in a single error message	2022-12-03 22:02:04 -08:00
Aliaksandr Valialkin	81400c80f0	lib/fs: remove logging redundant path values in a single error message	2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin	dc890ab80c	lib/backup: remove logging duplicate path values in a single error message	2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin	6910b1de2e	all: typo fix: the the -> the	2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin	0b8e7deabd	lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 ... Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message appears in the log and then upgrade to newer release.	2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin	8e9822bc7f	lib/storage: speed up search for data block for the given tsids ... Use binary search instead of linear scan for looking up the needed data block inside index block. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin	d8303845ef	lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC	2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin	d8d4d21d7a	lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts ... Move the common code into releasePartsToMerge() method and consistently use it throughout the code.	2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin	be6da5053f	lib/promscrape: optimize service discovery speed ... - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin	2524d94fe1	lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels	2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin	027ab74efb	lib/httpserver: link to url format docs in error message emitted on parse error for the provided url path ... This should help users identifying and fixing improperly set up urls. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402	2022-11-28 18:49:03 -08:00
Aliaksandr Valialkin	8ce5b095b7	lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:34 -08:00
匠心零度	d4808d5b84	lib/storage: remove extra error check (#3396)	2022-11-28 17:07:11 +01:00
Aliaksandr Valialkin	c5eebaffd8	app/{vminsert,vmagent}: follow-up after 53a63c6c4c ... Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:42:38 -08:00
Zakhar Bessarab	e407e7243a	{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348) ... * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * app/vmselect: fix error message * {app/vmstorage,app/vmselect}: fix error messages * app/vmselect: change log level for error handling * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-25 10:32:45 -08:00
Roman Khavronenko	d1169c1559	vmagent: expose metrics for tracking config state (#3375) ... Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin	c33bcae457	lib/promscrape/discovery/gce: do not pass filter arg when discovering zones ... The filter arg isn't supported by zones API in GCE. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202	2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin	48dda17e36	lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration ... While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes. This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-17 21:55:11 +02:00
Aliaksandr Valialkin	047fe3ee67	lib/promscrape: add a benchmark for internLabelStrings()	2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin	b0b8f05fa4	lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() ... The issue has been introduced in 58b40f514c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin	c8d7e1312c	lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio ... Reduce the default value of this flag from 1% to 0.2% after 71335e6024 This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:41:37 +02:00
Aliaksandr Valialkin	06300fe9b8	lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf ... Previously indexBlock.bhs pointed to the buffer, which could be changed over time. This could result in incorrect time series search over time. This is a follow-up for 58b40f514c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin	454060fd78	lib/mergeset: remove string allocation and copying when unmarshaling blockHeader ... This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin	a5a0fa75bf	lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1% ... It has been appeared that some production workloads could suffer for some time after every reset of the previous cache when it gets less than 5% of requests after the needed item isn't found in the current cache. This could result in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM usage needed for reading, unpacking and caching the missed data from disk. This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%. This should reduce the possible negative impact after each cache reset by at least 5x, while reducing the total memory used by caches. This is a follow-up for d906d8573e	2022-11-10 12:38:53 +02:00
Aliaksandr Valialkin	24213eaeba	lib/promscrape: add more cases to TestAddRowToTimeseries ... This is a follow-up for 16fdd2af8a	2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL	87375b004a	test(golang): add test to function addRowToTimeseries (#3282) ... Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>	2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin	2091693f16	lib/protoparser/opentsdb: follow-up after 04b0e4e7bf ... - Simplify the parser code to be less error prone - Document the change - Add a test for OpenTSDB put line with trailing whitespace without tags Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290	2022-11-09 15:36:15 +02:00
Roman Khavronenko	71dfe4d697	protoparser/opentsdb: allow lines without tags (#3303) ... According to http://opentsdb.net/docs/build/html/api_telnet/put.html "At least one tag pair must be present". However, in VictoriaMetrics datamodel tags aren't required. This could be confusing for users. Allowing accept lines without tags seems to do no harm. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin	abf7e4e72f	lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does ... See https://github.com/prometheus/prometheus/pull/11482	2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin	d3035b1ca1	lib/storage: follow-up for 790768f20b ... - Document the bugfix at docs/CHANGELOG.md - Simplify the bugfix a bit	2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin	be78950011	lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8	2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin	9d901ee55a	lib/envtemplate: allow non-env var names inside "%{ ... }"	2022-11-07 13:16:00 +02:00