Commit Graph

74 Commits

Author SHA1 Message Date
JAYICE
a696ef21df
docs: correct the wrong description of vector support in the docs (#7566)
### Describe Your Changes

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7557

Vector only supports OpenTelemetry as a source.
2024-11-18 16:30:43 +01:00
Crazygit
5eb6a0f9da
docs/vlogs: fixes typo for VL-Extra-Field
`VL-Extra-Field` should changed to `VL-Extra-Fields` according to code
2024-11-14 11:20:30 +01:00
Aliaksandr Valialkin
b7c290ea5e
docs/VictoriaLogs/CHANGELOG.md: refer to the issue related to adding fields to Syslog logs
This is a follow-up for cd60a4c589
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354
2024-11-08 21:50:14 +01:00
Aliaksandr Valialkin
4f0bec6f03
app/vlinsert/syslog: allow changing the default set of log fields to use as stream fields during syslog data ingestion
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7488
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7480

See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#stream-fields
2024-11-08 21:21:08 +01:00
Aliaksandr Valialkin
cd60a4c589
app/vlinsert/syslog: add an ability to drop and add fields during data ingestion via Syslog protocol
See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#dropping-fields
and https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#adding-extra-fields
2024-11-08 20:57:59 +01:00
Andrii Chubatiuk
30dd4cdc0d
docs: add make command for docs version update (#7430)
### Describe Your Changes

added make target, which updates `{{% available_from "#" %}}` shortcode
to `{{% available_from "$(PKG_TAG)" %}}` if PKG_TAG matches expression
`v.*`. `{{% available_from %}}` shortcode was introduced in
https://github.com/VictoriaMetrics/vmdocs/pull/89 to show a reference to
a version in a changelog since which a feature was introduced

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7376

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-08 16:35:08 +01:00
Andrii Chubatiuk
e0930687f1
vlinsert: support datadog logs
This commit adds the following changes:

- Added support to push datadog logs with examples of how to ingest data
using Vector and Fluentbit
- Updated VictoriaLogs examples directory structure to have single
container image for victorialogs, agent (fluentbit, vector, etc) but
multiple configurations for different protocols

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632
2024-11-05 16:52:35 +01:00
Aliaksandr Valialkin
37a58677fa
docs/VictoriaLogs/data-ingestion: mention Grafana Agent and Grafana Alloy additionally to Promtail as log collectors for Grafana Loki 2024-11-02 11:36:22 +01:00
Aliaksandr Valialkin
4478e48eb6
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445
2024-11-01 20:06:17 +01:00
Aliaksandr Valialkin
96466562b6
docs/VictoriaLogs/data-ingestion/README.md: formatting fixes after 8b36529b32
- Remove leading whitespace from the first lines in 'HTTP parameters' chapter.
  This whitespace isn't needed for the markdown formatting.

- Add leading whitespace for the second sentence in the list bullet describing AccountID and ProjectID HTTP headers.
  This fixes markdown formatting for this list bullet.
2024-10-30 14:23:54 +01:00
Aliaksandr Valialkin
ed73f8350b
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.
2024-10-30 14:17:33 +01:00
Andrii Chubatiuk
7e60afb6fc
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:36:33 +01:00
n4mine
f060b67da5
docs: fix typo in docs/VictoriaLogs/data-ingestion/Vector.md (#7222)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-10 09:45:30 +02:00
n4mine
344d61da79
docs: fix typo in docs/VictoriaLogs/data-ingestion/Vector.md (#7208)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-09 00:32:36 -07:00
Aliaksandr Valialkin
806bc2ac58
app/vlinsert: support unix timestamps in seconds and milliseconds in JSON stream data ingestion API 2024-09-28 21:56:50 +02:00
Aliaksandr Valialkin
7d7d7c03bc
app/vlinsert: accept unix timestamp in seconds additionally to milliseconds at ElasticSearch bulk API
Timestamps in seconds are sometimes used for data ingestion via ElasticSearch bulk API
2024-09-28 21:19:54 +02:00
Andrii Chubatiuk
05a64a8c14
victorialogs: marked fluentd support in roadmap, added syslog example (#7098)
### Describe Your Changes

Marked fluentd in victorialogs roadmap
Added fluentd syslog example setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-27 14:38:39 +02:00
Aliaksandr Valialkin
037652d5ae
app/vlinsert: support _time field without timezone information during data ingestion
Use local timezone of the host server in this case. The timezone can be overridden
with TZ environment variable if needed.

While at it, allow using whitespace instead of T as a delimiter between data and time
in the ingested _time field. For example, '2024-09-20 10:20:30' is now accepted
during data ingestion. This is valid ISO8601 format, which is used by some log shippers,
so it should be supported. This format is also known as SQL datetime format.

Also assume local time zone when time without timezone information is passed to querying APIs.
Previously such a time was parsed in UTC timezone. Add `Z` to the end of the time string
if the old behaviour is preferred.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6721
2024-09-26 12:49:35 +02:00
Andrii Chubatiuk
f934f71708
docs/victorialogs/data-ingestion: removed FluentBit Elasticsearch from examples (#7093)
removed FluentBit Elasticsearch example from docs as custom headers are
not supported by elasticsearch output till
https://github.com/fluent/fluent-bit/pull/9416 is merged and released

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6985

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-25 18:43:26 +02:00
Andrii Chubatiuk
e75ae1b274
deployment: restructure victorialogs examples (#6971)
### Describe Your Changes

- Use common compose.yaml file for all victorialogs setups to set
version in a single place and override it on demand for each agent and
protocol
- Replaced multiple victorialogs instances in HA setup with single setup
with `deploy.replica` parameter set
- Added fluentd setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-25 18:33:26 +02:00
Zhu Jiekun
fea4433362
docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074)
### Describe Your Changes

By default, the `elasticsearchexporter` in OTel Collector puts the log
message under a field other than `_msg` (e.g., `Body`). Without
specifying via an HTTP header, those logs may not be queried correctly.
See also:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785.

This pull request updates the example configuration and notes for the
`elasticsearchexporter`.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-24 11:52:09 +02:00
Dima Lazerka
8207879fa3
docs: fixes misspelled typos
Also tried to make it catch "Authorisation" in the future, fixed a lot
of other misspells along the way, but didn't make it catch
"Authorisation" anyway.

- Fix misspelled "Authorization" header name
- Fix misspelled "organization"
- Fix more misspells
2024-09-13 12:14:24 +02:00
Artem Navoiev
ae5f98f46b
docs: fix links to docker
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 19:56:58 +02:00
Andrii Chubatiuk
01430a155c
vlinsert: added opentelemetry logs support
Commit adds the following changes:

* Adds support of OpenTelemetry logs for Victoria Logs with protobuf encoded messages

*  json encoding is not supported for the following reasons:
   - It brings a lot of fragile code, which works inefficiently.
   - json encoding is impossible to use with language SDK.

* splits metrics and logs structures at lib/protoparser/opentelemetry/pb package.

* adds docs with examples for opentelemetry logs.

---
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4839

Co-authored-by: AndrewChubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 20:12:05 +02:00
f41gh7
8b36529b32
follow-up after 1731c0eabf
* updates change log
* adds VL-Debug http header
* updates doc
* extracts only the first value of http headers for VL-Stream-Fields and VL-Ignore-Fields.
  It makes behaviour the same as Query string args. And allows to easily configure client applications.
  Since most of the client collectors don't support multi value headers.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 19:16:10 +02:00
Andrii Chubatiuk
1731c0eabf
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 17:43:26 +02:00
Andrii Chubatiuk
35d77a3bed
docs: updated guides structure, removed deprecated sort option (#6767)
### Describe Your Changes

* `sort` param is unused by the current website engine, and was present only for compatibility
with previous website engine. It is time to remove it as it makes no effect
* re-structure guides content into folders to simplify assets management

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 16:48:08 +02:00
Andrii Chubatiuk
2e16732fdb
docs: updated docs titles and links (#6741)
The changes are based on SEO report and supposed to improve
ranking and indexation by search engines by using prompt and unique titles
and by updating unreachable links.

It also updates links to have a simplified form and replaces relative links with absolute links
according to https://docs.victoriametrics.com/#documentation

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-08-06 15:54:52 +02:00
Andrii Chubatiuk
ce4cc4cbb2
view documentation locally (#6677)
### Describe Your Changes

- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 01:00:31 -07:00
Andrii Chubatiuk
332b7788fb
added _index.md for hugo, which point to README.md (#6686)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-23 17:30:49 +03:00
Aliaksandr Valialkin
1f96882367
docs: fix spellcheck errors found by make spellcheck
Thanks to the spellcheck Makefile rule added by @arkid15r at fabf0b928e
2024-07-17 18:14:20 +02:00
Aliaksandr Valialkin
ba6f82069f
app/vlinsert/syslog: add an ability to use log ingestion time as the _time field 2024-07-02 01:38:19 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin
7d026b4655
docs/VictoriaLogs/data-ingestion/README.md: document that Rsyslog and Syslog-ng can be used for ingesting logs into VictoriaLogs 2024-06-25 15:15:46 +02:00
Aliaksandr Valialkin
3eacd43fff
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
Aliaksandr Valialkin
e498fa6960
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:34 +02:00
Aliaksandr Valialkin
478468e6cd
app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425 2024-06-17 22:28:26 +02:00
Aliaksandr Valialkin
b5a206cea1
docs/VictoriaLogs: mention which collectors support sending logs to VictoriaLogs in syslog format 2024-06-17 13:13:43 +02:00
Aliaksandr Valialkin
2b6a634ec0
lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
Artem Navoiev
d1baa41b6a
docs: add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-06-14 12:32:12 +02:00
Arkadii Yakovets
c740a8042e
docs: fix docs/ and README.md spelling errors (#6362)
Fixes `docs/` and `README.md` typos and errors.

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-06-03 10:04:13 +02:00
Aliaksandr Valialkin
e2590f0485
lib/logstorage: work-in-progress 2024-05-25 00:30:58 +02:00
Aliaksandr Valialkin
9dbd0f9085
lib/logstorage: initial implementation of pipes in LogsQL
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:33:31 +02:00
hagen1778
4f55aa29db
docs: mention HTTP sink configuration example for Vector
Follow-up 16eeb4e

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-16 14:07:16 +02:00
Devin Buhl
16eeb4eb33
victorialogs: mention vector supports http/json stream (#6114)
https://github.com/vectordotdev/vector/issues/18883#issuecomment-1771424716
2024-04-16 13:52:03 +02:00
Artem Navoiev
c300a636d6
docs: change ndjson links to https://jsonlines.org/ as original one was hacked (#5782)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-02-08 16:06:31 +01:00
hagen1778
6c6c2c185f
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-27 19:29:11 +01:00
Aliaksandr Valialkin
6f98b9c221
Revert "docs victorialogs use relative links"
This reverts commit 3d7a77bf82.

Reason for revert: relative links do not work properly at GitHub code
and at GitHub wiki. For example, the following page contains broken links
before reverting this commit:

https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/VictoriaLogs/CHANGELOG.md

It is always better to use absolute links thank relative links, since the page contents
can be copy-n-pasted to other pages, which are located in vastly different directories,
and all the links will remain working.
2023-10-16 13:40:43 +02:00
Artem Navoiev
f5c46b8176
docs fix bad links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-14 14:56:06 +02:00
Artem Navoiev
3d7a77bf82 docs victorialogs use relative links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-14 05:42:29 -07:00