Commit Graph

21 Commits

Author SHA1 Message Date
Nikolay
5514b5d552
app/vminsert: allow parsing tenant id from (#4144)
VictoriaMetrics_ProjectID and VictoriaMetrics_AccountID labels.
It should help to migrate for new labels vm_account_id vm_project_id without service downtime
2023-05-16 08:16:37 -07:00
Nikolay
82e2f19bc2
app/vminsert: correctly allocate buffer for storagenodes (#554)
in case of dynamic discovery number of nodes may change and we have to allocate new buffer for this case
otherwise vminsert may panic
2023-05-08 08:57:15 -07:00
Aliaksandr Valialkin
8540dd669b
app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket
This should prevent from panics when the list of discovered vmstorage nodes changes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
2022-11-09 11:45:24 +02:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin
dceca7e864
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package
This is a follow-up for fe2269b999
2022-06-21 20:27:30 +03:00
Aliaksandr Valialkin
021ee53ba8
app/vminsert: improve re-routing logic in order to spread rows more evenly among the available storage nodes 2022-02-06 20:20:02 +02:00
Aliaksandr Valialkin
64b6f3f1c8
app/vminsert: fix uneven distribution of time series among storage nodes
Use distinct seed for distribution hash calculations on the second level of vminsert nodes.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672
2021-10-07 12:22:39 +03:00
Aliaksandr Valialkin
512addc608 app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
2021-03-31 23:27:21 +03:00
Aliaksandr Valialkin
c91ccce50c app/vminsert: fix relabeling for metrics ingested via Influx line protocol
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin
6ebac3ab63 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries
See https://victoriametrics.github.io/#relabeling
2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
7a61357b5d app/vminsert/netstorage: make sure that the the data is always replicated among -replicationFactor vmstorage nodes
Previously vminsert could write multiple copies of the data to a single vmstorage node when the ingestion rate
exceeds the maximum throughput for connections to vmstorage nodes.
2020-05-28 19:59:07 +03:00
Aliaksandr Valialkin
77e5165e7b app/vminsert: add -replicationFactor command-line flag for enabling data replication among available -storageNode instances 2020-05-27 17:29:44 +03:00
Aliaksandr Valialkin
75f2f3b09d app/vminsert/netstorage: improve ingestion performance when a single vmstorage node is slower than other vmstorage nodes
Previously the ingestion performance has been limited by the slowest vmstorage node.
Now vminsert should re-route data from the slowest vmstorage node to the remaining nodes.
2020-05-27 15:08:22 +03:00
Aliaksandr Valialkin
6edc33d9bb app/vminsert/netstorage: capture the first error instead of the last error when sending data to vmstorage
The first error has more chances to point to the real root cause of the issue.
2020-05-22 17:49:33 +03:00
Aliaksandr Valialkin
c81a89a8ed app/vminsert: add /api/v1/import handler
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6
2019-12-09 22:37:49 +02:00
Aliaksandr Valialkin
2f4c950fe9 app/vminsert/netstorage: send per-storageNode bufs to vmstorage nodes in parallel
This should improve the maximum ingestion throughput.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-11 14:50:19 +03:00
Aliaksandr Valialkin
c197641978 all: return 503 http error if service is temporarily unavailable
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin
5fcdb4a59a app/vminsert: improve handling of unhealthy vmstorage nodes
* Spread load evenly among remaining healthy nodes instead of hammering
  the next node after the unhealthy node.
* Make sure that the packet is flushed to storage node before returning success.
  Previously packets could stay in local buffers and thus lost on connection errors.
* Keep rows in the limited memory when all the storage nodes are unhealthy.
2019-06-09 00:42:36 +03:00
Aliaksandr Valialkin
24578b4bb1 all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00