From e6ec442e9671f7fb13aba1020d6f187074589931 Mon Sep 17 00:00:00 2001 From: Aliaksandr Valialkin Date: Fri, 14 May 2021 17:21:41 +0300 Subject: [PATCH] docs/Single-server-VictoriaMetrics.md: document how to reduce memory usage when importing too long JSON lines into VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1295 --- README.md | 2 ++ docs/Single-server-VictoriaMetrics.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index d71d5c9fb..a127a8f8f 100644 --- a/README.md +++ b/README.md @@ -957,6 +957,8 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. +VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines. + ### How to import CSV data diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index 6cc56b94c..5f21958f0 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -961,6 +961,8 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail. +VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines. + ### How to import CSV data