Aliaksandr Valialkin
4df7573858
lib/protoparser/csvimport: properly parse the last empty column in CSV line
...
Do not ignore the last empty column in CSV line.
While at it, properly parse CSV columns in single quotes, e.g. `'foo,bar',baz` is parsed as two columns - `foo,bar` and `baz`
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048
See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298
2023-05-12 16:59:50 -07:00
Aliaksandr Valialkin
6fd39e2000
Revert "lib/protoparser: fix skip csv line when metric can be collect from the line ( #4298 )"
...
This reverts commit 410ae99c2ehttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298
2023-05-12 16:59:11 -07:00
Dmytro Kozlov
714236c2d8
lib/protoparser: fix skip csv line when metric can be collect from the line ( #4298 )
...
* lib/protoparser: fix skip csv line when metric can be collect from the line
* lib/protoparser: fix comment
2023-05-12 15:53:52 -07:00
Aliaksandr Valialkin
1801fa6c5c
lib/protoparser/csvimport: extract stream parsing code into a separate stream package
...
This is a follow-up for 057698f7fb
2023-02-13 10:26:29 -08:00
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
...
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.
It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:07:16 -08:00
Roman Khavronenko
c22122212d
csvimport: support empty values ( #3565 )
...
Before, if the imported line contained multiple metrics and one
or more of them had an empty values - the whole line was ignored.
Now, only metrics with empty values are ignored, and the rest
of the metrics are accepted successfully.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-29 11:56:02 -08:00
Aliaksandr Valialkin
eab8ebbe11
all: make fmt via the upcoming Go1.19
2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin
88c2631320
lib/protoparser: remove superflowous memory allocations during protocol parsing
2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin
ebaf68bcb0
lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error
...
This is a follow-up for af90c3c43b
2021-06-14 15:20:38 +03:00
Aliaksandr Valialkin
1a237c6903
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
7f3e884a31
all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9
2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin
22c1e29284
lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols
...
The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-13 13:05:34 +02:00
Aliaksandr Valialkin
ed724d25ba
lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics
2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
1481d6d8ff
lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer
2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin
8df33bd5c1
app/{vminsert,vmagent}: improve data ingestion speed over a single connection
...
Process data obtianed from a single connection on all the available CPU cores.
2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
7072db75cb
lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server
...
This should reduce syscall overhead when reading big amounts of data
2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin
cd87ca303f
lib/protoparser: report more errors for incorrect timestamps and/or values
...
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin
76064ba9e7
Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer
2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin
2012e294d1
properly calculate readCalls
2020-07-10 12:01:05 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
661cfb03e2
lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data
...
These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols
2020-04-10 12:44:46 +03:00
Aliaksandr Valialkin
fa7910fba1
lib/protoparser/csvimport: add missing metric vm_rows_invalid_total{type="csvimport"}
2020-03-12 15:28:10 +02:00
Aliaksandr Valialkin
2f0a36044c
app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv
2020-03-10 21:17:40 +02:00
