Aliaksandr Valialkin
87390443d1
lib/protoparser/datadog: do not re-use previously parsed field values if they are missing in the currently parsed message
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-11 13:09:41 -08:00
Pedro Gonçalves
84cfe4fcaf
Datadog - Add device as a tag if it's present as a field in the series object ( #3431 )
...
* Datadog - Add device as a tag if it's present as a field in the series object
* address PR comments
2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
...
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:42:38 -08:00
Aliaksandr Valialkin
2091693f16
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
...
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:36:15 +02:00
Roman Khavronenko
71dfe4d697
protoparser/opentsdb: allow lines without tags ( #3303 )
...
According to http://opentsdb.net/docs/build/html/api_telnet/put.html
"At least one tag pair must be present".
However, in VictoriaMetrics datamodel tags aren't required.
This could be confusing for users. Allowing accept lines without
tags seems to do no harm.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin
e4e2d1fcde
lib/protoparser/clusternative: allocate unmarshalWork after reading the data from input connection
...
This shortens the time when unmarshalWork is in use. This also reduces the number of unmarshalWork
objects in the pool, and its memory usage.
2022-10-18 00:24:04 +03:00
Aliaksandr Valialkin
481ca746ba
lib/protoparser/clusternative: reuse unmarshalWork in order to reduce memory allocations
2022-10-18 00:06:56 +03:00
Aliaksandr Valialkin
958c1f291c
app: follow-up after ec04fcac93
...
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:54:15 +03:00
Dmytro Kozlov
4064db27a8
Properly parse json when export import metric ( #3180 )
...
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API
* app/vmselect: update convert function
* app/vmselect: export null if `math.IsNaN(v)`
* app/vmselect: get float from json
* lib/protoparser: add test
* docs: add change log
* lib/protoparser: make export import api compatible
2022-10-06 14:54:14 +03:00
Aliaksandr Valialkin
4afa25fb38
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed
2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
9c6c691471
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
...
This is a follow-up for 7c2474dac7https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
2b98f2bc1a
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102
See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
...
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin
eab8ebbe11
all: make fmt via the upcoming Go1.19
2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin
95add1e8e4
app/{vmagent,vminsert}: follow-up after d19e46de55
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839
2022-07-07 01:32:11 +03:00
Aliaksandr Valialkin
e3bf464f11
lib/protoparser/native: follow-up after fe01f4803d
2022-04-11 19:27:53 +03:00
Nikolay
39225fc809
lib/protoparser/native: fixes parseStream dead-lock ( #2423 )
...
previously, if native block cannot be unmarshaled, wg.Done wasn't called by unmarshal work.
It leads to connection blocking and possible dead-lock at client side
2022-04-11 19:27:51 +03:00
Aliaksandr Valialkin
88c2631320
lib/protoparser: remove superflowous memory allocations during protocol parsing
2022-04-06 14:00:50 +03:00
Nikolay
7eb49d204f
vmctl verify-blocks command ( #2390 )
...
* lib/protoparser: changes ParseStream for native format
uses reader instead of http.Request
updates app/vmagent and app/vmagent method usage
* app/vmctl: add verify-block subcommand
it allows to check exported from VictoriaMetrics data block in native format
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362
Update app/vmctl/README.md
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-04-05 17:46:36 +03:00
Aliaksandr Valialkin
8752cce157
app/vminsert: reduce the max packet size, which vminsert can send to vmstorage
...
This reduces the max memory usage for vminsert and vmstorage under heavy ingestion rate
by up to 50% on production workload
2022-04-05 15:39:58 +03:00
Aliaksandr Valialkin
5f266370c5
all: follow-up after 4bdd10ab90
...
Properly use new bytesutil.Resize* functions
2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin
02b2bfcff3
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations
...
Follow-up for f4989edd96https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
6232eaa938
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions
...
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().
Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.
Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin
d1289383eb
lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp
2021-12-02 13:44:01 +02:00
Aliaksandr Valialkin
37a2bea072
lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865
2021-12-02 13:33:50 +02:00
Aliaksandr Valialkin
975498d402
lib/protoparser/prometheus: follow-up for 8e338632a3
...
Do not spend CPU time on error message formatting if error logger is disabled
2021-11-30 00:51:15 +02:00
Nikolay
40f0726147
Changes unmarshallRow logger to noop for getRowsDiff ( #1835 )
2021-11-30 00:51:14 +02:00
Aliaksandr Valialkin
4ad397188e
lib/protoparser: do not log connection reset by peer error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections
...
This error is expected, so there is no need in spamming the log with this error.
2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin
6873d6d893
lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp
2021-10-28 12:48:34 +03:00
Aliaksandr Valialkin
92d01db85a
lib/protoparser/prometheus: optimize GetRowsDiff() function
...
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 ,
since the provided profile shows that the majority of CPU and memory is spent in this function
during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.
2021-10-27 18:55:25 +03:00
Aliaksandr Valialkin
16f1aaf0b5
lib/protoparser/prometheus: add a benchmark for GetRowsDiff
2021-10-27 18:55:23 +03:00
Nikolay
83e1dfccba
adds tab as second separator for graphite text protocol ( #1733 )
...
* adds tab as second separator for graphite text protocol
* changes indexFunc for indexAny
* Update lib/protoparser/graphite/parser_test.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-22 12:29:27 +03:00
Aliaksandr Valialkin
a47754b689
lib/protoparser/clusternative: typo fix after 4fddcf4c83
2021-10-08 15:38:47 +03:00
Aliaksandr Valialkin
4fddcf4c83
app/{vminsert,vmstorage}: follow-up after a171916ef5
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269
2021-10-08 14:09:51 +03:00
Nikolay
a171916ef5
Adds read-only mode for vmstorage node ( #1680 )
...
* adds read-only mode for vmstorage
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269
* changes order a bit
* moves isFreeDiskLimitReached var to storage struct
renames functions to be consistent
change protoparser api - with optional storage limit check for given openned storage
* renames freeSpaceLimit to ReadOnly
2021-10-08 12:52:56 +03:00
Aliaksandr Valialkin
7b69d478ec
lib/protoparser: go fmt
2021-09-29 21:17:49 +03:00
Aliaksandr Valialkin
6167890d0e
lib/protoparser/prometheus: compare invalid Prometheus lines in full
2021-09-29 19:41:23 +03:00
Aliaksandr Valialkin
4e65bfcc00
app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API
...
See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206
2021-09-29 00:12:26 +03:00
Nikolay
dd53abf36d
changes protoparser apis for accepting reading from io.Reader ( #1624 )
...
adds InsertHandlerForReader apis to vmagent
2021-09-20 14:54:20 +03:00
Aliaksandr Valialkin
184e145570
docs: consistency renaming: Influx -> InfluxDB
2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin
09670479cd
lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric
...
Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics.
The main idea is to perform the calculations on scrape_series_added and series_limit
only if the set of metrics exposed by the target has been changed.
Scrape targets rarely change the set of exposed metrics,
so this optimization should reduce CPU usage in general case.
2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin
5aaaa686a4
lib/promscrape: send stale markers for disappeared metrics like Prometheus does
2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin
18d7adf731
lib/protoparser/opentsdb: follow-up after 8ee75ca45a
2021-08-29 11:50:01 +03:00
envzhu
00dddfe02f
lib/protoparser/opentsdb: accept multiple spaces between fields in a row as a deliminator. ( #1575 )
2021-08-29 11:50:00 +03:00
Aliaksandr Valialkin
61e483a01c
lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime
...
This is a follow-up for b84aea1e6e
2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin
b84aea1e6e
lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct)
...
This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores
2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin
ebaf68bcb0
lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error
...
This is a follow-up for af90c3c43b
2021-06-14 15:20:38 +03:00
faceair
2ea187e801
lib/protoparser: stop read when callback error ( #1380 )
2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin
074b11fa69
lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin
87d221f78a
lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error
...
This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
2021-06-14 12:21:21 +03:00
