Commit Graph

481 Commits

Author SHA1 Message Date
Andrii Chubatiuk
7e60afb6fc
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:36:33 +01:00
Fred Navruzov
b17fce3e4b
docs/vmanomaly-release-1.17.2 (#7322)
### Describe Your Changes

- release 1.17.2 updates
- added sections on logging and CLI args to docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-22 13:42:37 +03:00
f41gh7
50487823ab
deployment: bump VM to v1.105.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-10-21 23:42:30 +02:00
Fred Navruzov
7a538bbe78
docs/vmanomaly: release 1.17.1 (#7302)
### Describe Your Changes

docs/vmanomaly: release 1.17.1

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-18 21:07:24 +04:00
Aliaksandr Valialkin
025eec2cb0
deployment: update VictoriaLogs from v0.36.0-victorialogs to v0.37.0-victorialogs
Some checks are pending
CodeQL Go / Analyze (push) Waiting to run
CodeQL JS/TS / Analyze (push) Waiting to run
main / lint (push) Waiting to run
main / test (test-full) (push) Blocked by required conditions
main / test (test-full-386) (push) Blocked by required conditions
main / test (test-pure) (push) Blocked by required conditions
publish-docs / Build (push) Waiting to run
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.37.0-victorialogs
2024-10-18 02:40:13 +02:00
Fred Navruzov
bc65c9f399
docs/vmanomaly: release v1.17.0 (#7285)
### Describe Your Changes

docs/vmanomaly: release v1.17.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-17 18:56:04 +03:00
Andrii Chubatiuk
3538869942
vlogs: added basic alerts (#7252)
### Describe Your Changes

Added basic VLogs alerts

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:33:06 +02:00
Aliaksandr Valialkin
9187ed0648
deployment: update VictoriaLogs Docker image from v0.35.0-victorialogs to v0.36.0-victorialogs 2024-10-16 20:10:05 +02:00
Lauri Tirkkonen
8fe41b2b08
deployment/alerts: fix quoting on DiskRunsOutOfSpace (#7234)
### Describe Your Changes

there's an extra `"` at the end of the dashboard url for this alert;
remove it by making the quoting consistent with other alerts in this
file.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Lauri Tirkkonen <lauri@hacktheplanet.fi>
2024-10-11 00:44:18 -07:00
Aliaksandr Valialkin
baa87b5b36
deployment: update VictoriaLogs from v0.34.0-victorialogs to v0.35.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.35.0-victorialogs
2024-10-09 16:17:51 +02:00
Fred Navruzov
f61d8c3ebb
docs/vmanomaly: updates for v1.16.3 (#7203)
Some checks are pending
publish-docs / Build (push) Waiting to run
### Describe Your Changes

docs/vmanomaly: updates for v1.16.3

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-08 19:24:56 +03:00
Aliaksandr Valialkin
03862368b5
deployment: update VictoriaLogs image tag from v0.33.0-victorialogs to v0.34.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.34.0-victorialogs
2024-10-08 12:30:27 +02:00
Zakhar Bessarab
eefae85450
vmagent: add support of HTTP2 client for Kubernetes SD (#7114)
### Describe Your Changes

Currently, vmagent always uses a separate `http.Client` for every group
watcher in Kubernetes SD. With a high number of group watchers this
leads to large amount of opened connections.

This PR adds 2 changes to address this:
- re-use of existing `http.Client` - in case `http.Client` is connecting
to the same API server and uses the same parameters it will be re-used
between group watchers
- HTTP2 support - this allows to reuse connections more efficiently due
to ability of using streaming via existing connections.

See this issue for the details and test results -
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5971

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-10-08 10:36:31 +02:00
hagen1778
07902baa8e
deployment: bump VM to v1.104.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 16:33:00 +02:00
Fred Navruzov
150ee902fd
docs/vmanomaly: patch release 1.16.1 (#7169)
### Describe Your Changes

`vmanomaly` patch release 1.16.1 updates

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 18:10:08 +04:00
Fred Navruzov
dde2a0cb25
docs/vmanomaly - update versions in examples (#7163)
### Describe Your Changes

update `vnanomaly` versions in examples

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 11:53:05 +03:00
Fred Navruzov
0e54cfe350
docs/vmanomaly - release 1.16.0 docs (#7159)
### Describe Your Changes

doc updates for vmanomaly v1.16.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 00:19:14 +02:00
Aliaksandr Valialkin
b05fbee63d
deployment: update VictoriaLogs docker image from v0.32.1-victorialogs to v0.33.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.33.0-victorialogs
2024-10-01 13:57:56 +02:00
Aliaksandr Valialkin
aafa9262c5
deployment/docker: update VictoriaLogs from v0.32.0-victorialogs to v0.32.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.1-victorialogs
2024-09-30 14:35:40 +02:00
Aliaksandr Valialkin
2047ad20ef
deployment: update VictoriaLogs docker image from v0.31.0-victorialogs to v0.32.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.0-victorialogs
2024-09-29 14:54:21 +02:00
Andrii Chubatiuk
dfcab4a47f
deployment/victorialogs: rename not executable compose files (#7124)
### Describe Your Changes

Renamed base compose files to prevent envs to be created from them

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-29 09:54:45 +02:00
Roman Khavronenko
fd890a2771
deployment: separate datasource between single and cluster versions (#7119)
Before, single and cluster deployments were provisioned with both
Grafana datasources: single and cluster. But this resulted into a
problem: single DS didn't work for cluster and vice versa. See
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7113

This PR splits datasource file into 2 files: single and cluster. Now,
these files are separately provisioned to single and cluster deployments
correspondingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-27 19:08:05 +02:00
Andrii Chubatiuk
00912bfa0f
docs/victorialogs: change fluentd input type in examples (#7123)
### Describe Your Changes

Use fluentd logging driver in examples to have enriched data in
VictoriaLogs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-27 15:09:36 +02:00
Andrii Chubatiuk
05a64a8c14
victorialogs: marked fluentd support in roadmap, added syslog example (#7098)
### Describe Your Changes

Marked fluentd in victorialogs roadmap
Added fluentd syslog example setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-27 14:38:39 +02:00
Aliaksandr Valialkin
86c0eb816c
deployment: update VictoriaLogs docker image from v0.30.1-victorialogs to v0.31.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.31.0-victorialogs
2024-09-27 13:58:54 +02:00
Aliaksandr Valialkin
b670b0e9ff
deployment: update VictoriaLogs docker image tag from v0.30.0-victorialogs to v0.30.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.1-victorialogs
2024-09-27 11:26:15 +02:00
Aliaksandr Valialkin
6f1fde24dc
deployment: update VictoriaLogs docker image from v0.29.0-victorialogs to v0.30.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.0-victorialogs
2024-09-27 09:34:09 +02:00
Andrii Chubatiuk
e75ae1b274
deployment: restructure victorialogs examples (#6971)
### Describe Your Changes

- Use common compose.yaml file for all victorialogs setups to set
version in a single place and override it on demand for each agent and
protocol
- Replaced multiple victorialogs instances in HA setup with single setup
with `deploy.replica` parameter set
- Added fluentd setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-25 18:33:26 +02:00
Dmytro Kozlov
91b28d0527
deployment/docker: update grafana datasources to the latest version (#7083)
### Describe Your Changes

Updated grafana plugins to the latest releases 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-24 11:51:18 +02:00
hagen1778
a5c002edef
deployment/alerts: fix copy&paste typo in TooHighGoroutineSchedulingLatency
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-24 11:48:19 +02:00
Roman Khavronenko
4d0b41e63b
deployment: add panel and alerts for displying go scheduler latency (#7078)
The panel and alerting rule should help to understand whether VM
component doesn't have enough CPU resources or gets throttled. The alert
is applicable for all VM components.
The panel was added to vmalert, vmagent, vmsingle, vm clusert and
victorialogs dashes.

-------------------

This alerting rule should have help us identify resource shortage for
sandbox vmagent - see [this
link](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.range_input=23d13h25m25s424ms&g0.end_input=2024-09-23T14%3A11%3A00&g0.relative_time=none&g0.tab=0&g0.expr=histogram_quantile%280.99%2C+sum%28rate%28go_sched_latencies_seconds_bucket%7Bjob%3D%22vmagent-monitoring-vmagent%22%7D%5B5m%5D%29%29+by+%28le%2C+job%2C+instance%29%29+%3E+0.1)
for example. We weren't aware of resource shortage, because VM metrics
assumed this vmagent had 1vCPU while in fact its limit was 0.2vCPU.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-23 16:54:42 +02:00
Thomas Danielsson
69b976f08e
docs: typo filebeat -> fluent-bit (#7000)
### Describe Your Changes

Fix a typo; `filebeat.yml` -> `fluent-bit.conf`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Thomas Danielsson <thomas@elajt.se>
2024-09-19 14:33:12 +02:00
Thomas Danielsson
75cc7922c3
typo: added missing '{' (#7044)
### Describe Your Changes

Added missing `{` in vmalert rule.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-19 00:11:30 -07:00
Artem Navoiev
277fed9990
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 14:33:05 -07:00
Artem Navoiev
47cd7bb4c1
docs: update roadmap, mark telegraf and otel as done and link the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 17:35:14 +02:00
Aliaksandr Valialkin
0ed835026f
deployment: update VictoriaLogs from v0.28.0-victorialogs to v0.29.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.29.0-victorialogs
2024-09-08 21:23:16 +02:00
Aliaksandr Valialkin
4fbdde5852
deployment/docker: update base Alpine docker image from 3.20.2 to 3.20.3
See https://alpinelinux.org/posts/Alpine-3.17.10-3.18.9-3.19.4-3.20.3-released.html
2024-09-08 19:26:48 +02:00
Aliaksandr Valialkin
5261a84119
deployment: update Go builder from Go1.23.0 to Go1.23.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.1+label%3ACherryPickApproved
2024-09-06 22:51:15 +02:00
Dmytro Kozlov
85acc8a7c4
deployment: upgrade datasource versions (#6935)
### Describe Your Changes

Upgraded victoriametrics and victorialogs data source versions.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-04 16:30:37 +02:00
Andrii Chubatiuk
01430a155c
vlinsert: added opentelemetry logs support
Commit adds the following changes:

* Adds support of OpenTelemetry logs for Victoria Logs with protobuf encoded messages

*  json encoding is not supported for the following reasons:
   - It brings a lot of fragile code, which works inefficiently.
   - json encoding is impossible to use with language SDK.

* splits metrics and logs structures at lib/protoparser/opentelemetry/pb package.

* adds docs with examples for opentelemetry logs.

---
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4839

Co-authored-by: AndrewChubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 20:12:05 +02:00
Andrii Chubatiuk
1731c0eabf
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 17:43:26 +02:00
Aliaksandr Valialkin
91456ab5bb
all: suppress InvalidDefaultArgInFrom warning emitted by docker build when building Docker packages via make package-* command
Recent versions of `docker build` started generating the InvalidDefaultArgInFrom warning if Dockerfile contains
an ARG without default value. While this warning doesn't affect building Docker packages via `make package-*` commands,
it is better suppressing the warning, so it doesn't clutter `make package-*` output with the noise,
which can hide real issues in the future.
2024-09-03 14:00:28 +02:00
Marco Maurer (-Kilchhofer)
f17fca718d
deployment/alerts: add missing description for VMagent alerts (#6921)
The 3 alerts for VMagent:
- `RejectedRemoteWriteDataBlocksAreDropped`
- `TooManyScrapeErrors`
- `TooManyWriteErrors`

missed the description annotation.
I moved the summary to description and added a generic summary to these
alerts.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Marco Maurer <marco.kilchhofer@gmail.com>
2024-09-03 10:26:19 +02:00
hagen1778
b58a10c389
deployment: bump VM version to 1.103.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-29 15:13:19 +02:00
Roman Khavronenko
1b9f3b39b4
deployment/docker: update Go builder from Go1.22.5 to Go1.23.0 (#6861)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 23:55:50 +02:00
f41gh7
b85e3f8bbd
deployment: update compose images to v1.102.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 11:29:12 +02:00
Hui Wang
b515a7b69b
security: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2 (#6684)
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fix for:
OpenSSL CVE-2024-5535
2024-07-23 13:20:06 +02:00
Aliaksandr Valialkin
45794de0f9
deployment: update VictoriaMetrics Docker image from v1.101.0 to v1.102.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0
2024-07-17 22:34:48 +02:00
Aliaksandr Valialkin
3224a12e25
Revert "deployment: build image for vmagent streamaggr benchmark (#6515)"
This reverts commit 6b128da811.

Reason for revert: this complicates and slows down CI/CD without giving significant benefits in return.

The idea of automatic building, publishing and deploying Docker images to our playground on every pull request
and commit isn't very bright because of the following reasons:

- It slows down CI/CD pipeline
- It increases costs on CPU time spent at CI/CD pipeline
- It contradicts goal #7 at https://docs.victoriametrics.com/goals/#goals and non-goal #8 at https://docs.victoriametrics.com/goals/#non-goals

The previous workflow was much better - if we need to deploy some new Docker image at playground or staging environment,
then just __manually__ build and deploy the needed Docker image there. If the manual process requires making too many
steps, then think on how to automate these steps into a single Makefile command.

Updates https://github.com/VictoriaMetrics/ops/pull/1297
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6515
2024-07-16 13:27:06 +02:00
Aliaksandr Valialkin
b173f24041
docs/VictoriaLogs/CHANGELOG.md: cut v0.28.0-victorialogs release 2024-07-10 03:08:16 +02:00