115 Commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	cba820e390	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin	0b2086b7a5	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin	91b3482894	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin	b542e50680	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin	432187ac3b	app/vminsert: add /-/reload handler in the same way as for vmagent	2020-04-30 02:15:39 +03:00
Aliaksandr Valialkin	1fe66fb3cc	app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv	2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin	032c88561b	app/vminsert/prompush: limit memory usage by pushing promscrape data in smaller blocks	2020-03-03 19:58:54 +02:00
Aliaksandr Valialkin	6386aeb1e0	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP ... Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin	4c4448b66e	app/vminsert: add /targets handler, which exposes Prometheus targets defined in -promscrape.config file	2020-02-25 18:13:11 +02:00
Aliaksandr Valialkin	04762344c6	app/vmagent: initial implementation for vmagent	2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin	42864bb52f	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin	371e86194d	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin	6456c93dbb	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin	227cf53ef9	app/vminsert: increase default value for -insert.maxQueueDuration from 30s to 60s ... This should help catching up with high ingestion rate after VictoriaMetrics restart.	2020-01-18 14:39:36 +02:00
Aliaksandr Valialkin	c9472e4f3a	all: use github.com/klauspost/compress/gzip instead of compress/gzip ... `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin	d88725f133	app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects	2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin	0d7505b00e	all: mention command-line flags used for limiting the incoming request size in error messages ... This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin	4f0a645f77	vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5 ... This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.	2020-01-10 23:15:15 +02:00
Aliaksandr Valialkin	d03827c57d	app/vminsert: return StatusNoContent http response for /api/v1/import to be consistent with other insert handlers	2019-12-19 01:21:54 +02:00
Aliaksandr Valialkin	bc645152cb	app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin	68e1cf8942	app/vminsert: add /api/v1/import handler ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6	2019-12-09 20:59:04 +02:00
Aliaksandr Valialkin	8501b4a48d	app/vminsert: consistency renaming for counters	2019-12-09 16:43:10 +02:00
Aliaksandr Valialkin	293b541784	make vendor-update	2019-12-07 23:10:16 +02:00
Aliaksandr Valialkin	84b57e8974	app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689	2019-12-07 23:00:40 +02:00
Aliaksandr Valialkin	811b7a8303	app/vminsert/influx: allow empty measurement in Influx line protocol ... In this case metric names are mapped directly from field names without any prefixes.	2019-11-30 23:18:41 +02:00
Aliaksandr Valialkin	e83fe938c8	all: make fmt	2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin	97ce4e03a5	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin	d32f88c378	app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls	2019-09-19 22:15:59 +03:00
Aliaksandr Valialkin	fb909cf710	app/vminsert/influx: set db label only if Influx line doesnt have db tag	2019-08-24 13:52:48 +03:00
Aliaksandr Valialkin	9c9221d1b2	app/vminsert: skip empty tags	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	70ca018a57	app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	4266091e4f	app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	8001d29b6e	app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	9d3f1fcbb9	app/vminsert/influx: skip invalid rows and continue parsing the remaining rows ... Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	ba7b3806be	app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life ... The prefious report with escaped newline chars in influx line protocol was false alarm.	2019-08-23 18:42:05 +03:00
Aliaksandr Valialkin	7fa88c6efc	app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life	2019-08-23 18:35:41 +03:00
Aliaksandr Valialkin	4da34b11f8	app/vminsert/influx: handle \r\n aka crlf influx line endings from windows world ... Such lines exist in real life.	2019-08-23 18:28:49 +03:00
Aliaksandr Valialkin	a18317adbc	app/vminsert/influx: allow escaping newline char ... Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ , there are reports that such chars occur in real life	2019-08-23 15:14:46 +03:00
Aliaksandr Valialkin	44d7fc599d	app/vminsert/influx: skip comments starting with # in influx line protocol	2019-08-23 14:43:09 +03:00
Aliaksandr Valialkin	ac004665b5	all: return 503 http error if service is temporarily unavailable ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin	8c03a8c4b4	app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries	2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin	5f33fc8e46	app/vminsert: add ability to ingest data via HTTP OpenTSDB /api/put requests ... This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152 Thanks to nustinov@gmail.com for the initial pull request.	2019-08-22 12:28:32 +03:00
Aliaksandr Valialkin	ec8125606d	app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing put prefixes to each line	2019-08-21 19:14:47 +03:00
Aliaksandr Valialkin	5c83f8e203	app: add vm_concurrent_ metrics for visibility in concurrency limiters for vminsert and vmselect	2019-08-05 18:30:57 +03:00
Aliaksandr Valialkin	a9f39168d2	app/vminsert/influx: round automatically generated timestamp according to the given precision arg	2019-08-02 00:24:06 +03:00
Aliaksandr Valialkin	1eade9b358	app/vminsert: add vm_rows_per_insert summary metric ... This metric should help tuning batch sizes on clients writing data to VictoriaMetrics	2019-07-27 13:21:46 +03:00
Aliaksandr Valialkin	bf6cbb762c	app/vminsert: improve error messages for Influx, OpenTSDB and Graphite parsing ... Include in the error message the line which failed to parse. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/127	2019-07-26 22:08:52 +03:00
Aliaksandr Valialkin	575d2f0a91	app/vminsert: use netutil.TCPListener for collecting network-related metrics for Graphite and OpenTSDB TCP traffic	2019-07-15 22:58:00 +03:00
Jiri Tyr	e0fccc6c60	Change the default influxMeasurementFieldSeparator	2019-06-26 13:22:03 +03:00
Aliaksandr Valialkin	1f7d9a213a	app/vminsert: fix inifinite loop when reading two lines without newline in the end ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/82	2019-06-26 02:51:56 +03:00
Aliaksandr Valialkin	4b2523fb40	app/vminsert/opentsdb: remove unused const maxReadPacketSize; update https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:30:06 +03:00
Aliaksandr Valialkin	a8d4224828	app/vminsert/graphite: allow skipping timestamps in Graphite plaintext protocol ... In this case VictoriaMetrics uses the ingestion time as a timestamp.	2019-06-18 19:04:04 +03:00
Aliaksandr Valialkin	86d3d907a5	app/vminsert/influx: add -influxSkipSingleField flag for using {measurement} instead of {measurement}{separator}{field_name} for Influx lines with a single field ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/66	2019-06-17 19:05:57 +03:00
Aliaksandr Valialkin	269285848f	app/vminsert/influx: add -influxMeasurementFieldSeparator flag for the ability to change separator for {measurement}{separator}{field_name} metric name	2019-06-14 10:00:12 +03:00
Aliaksandr Valialkin	c0baa977cf	app/vminsert/concurrencylimiter: typo fix in the error message	2019-06-08 22:43:33 +03:00
Aliaksandr Valialkin	f4252f87e6	app/vminsert: really fix #60 ... ReadLinesBlock may accept dstBuf with non-zero length. In this case the last line without trailing newline isn't read. Fix this by comparing len(dstBuf) to 0 instead of its original length.	2019-06-07 23:37:03 +03:00
Aliaksandr Valialkin	0b78d228d2	app/vminsert: properly read trailing line without newline in the end ... This fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/60	2019-06-07 23:17:59 +03:00
Aliaksandr Valialkin	17f0a53068	app/vminsert: explain that /query request emulation is required for TSBS benchmark	2019-06-03 18:40:27 +03:00
Aliaksandr Valialkin	56b952c456	app/vminsert: add -maxConcurrentInserts command-line flag for limiting the number of concurrent inserts	2019-05-29 12:41:23 +03:00
Aliaksandr Valialkin	968d094524	app/vminsert: reduce memory usage for Influx, Graphite and OpenTSDB protocols ... Do not buffer per-connection data and just store it as it arrives	2019-05-28 18:47:23 +03:00
Aliaksandr Valialkin	e307a4d92c	lib/timerpool: use timer pool in concurrency limiters ... This should reduce the number of memory allocations in highly loaded system	2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin	0eae39daa7	app/vminsert: properly reset InsertCtx.mrs - they must be empty after Reset call	2019-05-28 16:08:01 +03:00
Aliaksandr Valialkin	feb6b203a4	app/vminsert/influx: try converting string values to numeric values, since Influx agents may send numeric values as strings ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/34	2019-05-26 22:11:19 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00