235 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	556484a52f	app/vminsert: return 200 OK status code when importing data in pushgateway format ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636	2023-01-15 13:26:09 -08:00
Aliaksandr Valialkin	b275983403	lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit ... Previously the -maxConcurrentInserts was limiting the number of established client connections, which write data to VictoriaMetrics. Some of these connections could be idle. Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting the number of such connections. So now the -maxConcurrentInserts command-line option limits the number of concurrently executed insert requests, not including idle connections. It is recommended removing -maxConcurrentInserts command-line option, since the default value for this option should work good for most cases.	2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin	7d996ced51	app/{vmagent,vminsert}/datadog: make the host label optional in DataDog data ingestion protocol ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432	2022-12-10 23:33:17 -08:00
Aliaksandr Valialkin	97b41e727c	lib/promscrape: implement target-level and metric-level relabel debugging ... Target-level debugging is performed by clicking the 'debug' link at the corresponding target on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page. Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407 See https://docs.victoriametrics.com/vmagent.html#relabel-debug	2022-12-10 02:25:56 -08:00
Pedro Gonçalves	84cfe4fcaf	Datadog - Add device as a tag if it's present as a field in the series object (#3431) ... * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin	3a25a4b1de	app/{vminsert,vmselect}: speed up TestInitStopNodes()	2022-12-03 23:53:14 -08:00
Aliaksandr Valialkin	c5eebaffd8	app/{vminsert,vmagent}: follow-up after 53a63c6c4c ... Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:42:38 -08:00
Aliaksandr Valialkin	afc35485c1	app/vminsert: add missing vm_relabel_config_* metrics after 03d88bc066 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345	2022-11-22 00:48:13 +02:00
Aliaksandr Valialkin	fe8d40f12c	app/{vminsert,vmselect}: test initialization with different number of storage nodes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329	2022-11-09 11:48:39 +02:00
Aliaksandr Valialkin	8540dd669b	app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket ... This should prevent from panics when the list of discovered vmstorage nodes changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329	2022-11-09 11:45:24 +02:00
Zakhar Bessarab	fd36c9b3b1	docs/{vminsert,vmselect}: clarify flag description for clusternativeListenAddr (#3314) ... docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`	2022-11-04 09:49:53 +01:00
Zakhar Bessarab	7afa67dba0	{app/vmselect,app/vminsert}: ensure -storageNodes flag has no empty values (#3291)	2022-11-01 14:54:55 +02:00
Aliaksandr Valialkin	09cf136154	app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes ... Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446	2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin	976bbe3677	app/{vminsert,vmselect}: limit the access to storageNodes to getStorageNodesBucket and setStorageNodesBucket functions ... This makes the code more maintainable and earier to test.	2022-10-28 11:41:55 +03:00
Aliaksandr Valialkin	4f53147ed4	app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop ... Previously netstorage.MustStop() call didn't free up all the resources, so the subsequent call to nestorage.Init() would panic. This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.	2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin	087393bcef	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places ... Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin	98bd843f3c	app/{vminsert,vmselect,vmstorage}: add package-{vminsert,vmselect,vmstorage}-{goarch} Makefile rules for building arch-specific docker images	2022-10-07 20:17:01 +03:00
Aliaksandr Valialkin	6f9ce3f6d6	lib/flagutil: rename Array to ArrayString ... This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:28:19 +03:00
Aliaksandr Valialkin	c8000c029a	app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints ... These labels are substituted by documented vm_account_id and vm_project_id labels. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels This is a follow up for 505d359b39	2022-09-30 18:06:00 +03:00
Nikolay	505d359b39	app/vminsert: allows parsing tenant id from labels (#3009) ... * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin	7f0b95b50a	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names ... Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for fcffdba9dc Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin	6a6cf9c590	app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113 Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146	2022-09-26 13:13:31 +03:00
Dmytro Kozlov	ec273eafef	app/{vmselect, vminsert}: fails with error when user defines equal addresses in the -storageNodes flag (#3082)	2022-09-08 21:17:58 +03:00
Aliaksandr Valialkin	dc58cb3fd8	app/vminsert/netstorage: pre-initialize the remaining throttled loggers ... This is a follow-up after 6c66804fd3	2022-08-04 18:34:42 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	f00a6bf837	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin	70b9925bf7	app: fix make publish-* after ed93330e66 ... Add missing `-linux` substring to built binary names for copying into Docker images	2022-07-14 11:01:34 +03:00
Aliaksandr Valialkin	da6c85a2f6	all: follow-up for d99ba3481b	2022-07-13 17:17:08 +03:00
Dmytro Kozlov	4e4def9df8	Rename release packages (#2810) ... * makefile: add os to each release file * makefile: update vmutils arm64 * makefile: update victoria-metrics release process * makefile: update publish with os * makefile: update publish with os * makefile: change tar library * update release logic * copy all releases * sort command by GOOS * rollback commands * rollback OSARCH * fix commands * cleanup * fix windows build * sort build by GOOS, update README.md	2022-07-13 17:11:01 +03:00
Aliaksandr Valialkin	95add1e8e4	app/{vmagent,vminsert}: follow-up after d19e46de55 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:32:11 +03:00
Pedro Gonçalves	af5f967307	app/vminsert: allow to ingest datadog metrics with simpler tags - not enforcing key:value (#2839)	2022-07-07 01:32:10 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them ... This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	dceca7e864	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package ... This is a follow-up for fe2269b999	2022-06-21 20:27:30 +03:00
Aliaksandr Valialkin	b28c6febf9	app/{vminsert,vmselect}: add -vmstorageDialTimeout command-line flag for tuning the maximum time needed for establishing connections to vmstorage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-20 15:17:34 +03:00
Dmytro Kozlov	f0cb3c4999	Added a stub for datadog endpoint (#2710) ... * Added a stub for datadog endpoint * Update app/vmagent/main.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-13 09:56:26 +03:00
Aliaksandr Valialkin	d830f43608	app/vminsert/main.go: add missing strings import after 0c5e199469	2022-06-07 15:42:23 +03:00
Aliaksandr Valialkin	2b72d21840	docs/CHANGELOG.md: document e755d0ec3f	2022-06-07 15:21:10 +03:00
elProxy	0c5e199469	Support legacy datadog agent (#2670) ... dd-agent v5 can issue some requests with trailing slashes. (e.g. 526559be73/ddagent.py (L303)) Trim trailing slashes for request on /datadog/ paths to accomodate for that. Co-authored-by: Pierre Rossi <pierre.rossi@schibsted.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-07 15:21:09 +03:00
Aliaksandr Valialkin	de143323d0	app/{vmagent,vminsert}: mention port 8089 instead of 8189 in the description for -influxListenAddr flag ... InfluxDB uses 8089 port for sending plain Influx line protocol data over TCP and UDP. See https://docs.influxdata.com/influxdb/v1.8/administration/ports/ This is a follow-up for 20cef877a1	2022-05-13 16:59:16 +03:00
Aliaksandr Valialkin	e613ca9ba8	app/vminsert/netstorage: re-route samples from readonly vmstorage nodes to healthy nodes if -dropSamplesOnOverload command-line flag is set	2022-05-07 01:40:02 +03:00
Aliaksandr Valialkin	925fa9a7de	docs/Cluster-VictoriaMetrics.md: make the description for -rpc.disableCompression command-line flag more clear	2022-05-06 16:24:56 +03:00
Nikolay	7eb49d204f	vmctl verify-blocks command (#2390) ... * lib/protoparser: changes ParseStream for native format uses reader instead of http.Request updates app/vmagent and app/vmagent method usage * app/vmctl: add verify-block subcommand it allows to check exported from VictoriaMetrics data block in native format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362 Update app/vmctl/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-04-05 17:46:36 +03:00
Aliaksandr Valialkin	8752cce157	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage ... This reduces the max memory usage for vminsert and vmstorage under heavy ingestion rate by up to 50% on production workload	2022-04-05 15:39:58 +03:00
Aliaksandr Valialkin	ce91456982	app/vminsert: add X-Influxdb-Version response header for InfluxDB API requests ... This is needed for some clients, which expect this header. See https://github.com/ntop/ntopng/issues/5449#issuecomment-1005347597	2022-02-17 12:49:24 +02:00
Eugene	8e1c78d5ab	Add -influxDBLabel parameter (#2203)	2022-02-16 23:25:01 +02:00
Aliaksandr Valialkin	755e26e67b	app/vminsert/netstorage: throttle warning logs, which can be too verbose when vminsert cannot send data to vmstorage nodes	2022-02-07 15:39:06 +02:00
Aliaksandr Valialkin	9fb5ce5fb6	app/vminsert/netstorage: log vmstorage addr, which cannot accept new samples due to overload and/or unavailability	2022-02-07 15:03:05 +02:00
Aliaksandr Valialkin	5aee6eb406	app/vminsert/netstorage: tune re-routing algorithm further	2022-02-07 14:35:39 +02:00
Aliaksandr Valialkin	1a5546006d	app/vminsert/netstorage: typo fix after 021ee53ba8	2022-02-07 13:20:43 +02:00
Aliaksandr Valialkin	c738739494	app/vminsert: add -dropSamplesOnOverload command-line flag ... Drop incoming samples if the destination vmstorage node is unavailable and/or accepts data at slower rate than other vmstorage nodes	2022-02-07 12:32:18 +02:00