Commit Graph

83 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
7a44614e0b
lib/logstorage: add len pipe for calculating byte length of log field values
(cherry picked from commit 364f084b43)
2024-10-04 10:42:51 +02:00
Aliaksandr Valialkin
04c73d54d4
lib/logstorage: support order alias for sort pipe
Now the following queries are equivalents:

    _time:5s | sort by (_time)

    _time:5s | order by (_time)

This is needed for convenience, since `order by` is commonly used in other query languages such as SQL.
2024-09-29 09:52:18 +02:00
Aliaksandr Valialkin
1a6313ca68
lib/logstorage: allow using - instead of ! as a shorthand for NOT operator in LogsQL 2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
b60cb98377
lib/logstorage: support skipping _stream: prefix for stream filters
'_stream:{...}' can be written as '{...}'

This simplifies writing queries with stream filters, and makes them more familier to Loki users.
2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
e83dc8b30a
docs/VictoriaLogs/LogsQL.md: use proper heading for blocks_count pipe docs
All the links in docs assume that the heading for pipe docs should end with `pipe` word.
This fixes broken links.

This is a follow-up for e9950f6307
2024-09-27 09:46:08 +02:00
Aliaksandr Valialkin
4d27933041
app/vlinsert: support _time field without timezone information during data ingestion
Use local timezone of the host server in this case. The timezone can be overridden
with TZ environment variable if needed.

While at it, allow using whitespace instead of T as a delimiter between data and time
in the ingested _time field. For example, '2024-09-20 10:20:30' is now accepted
during data ingestion. This is valid ISO8601 format, which is used by some log shippers,
so it should be supported. This format is also known as SQL datetime format.

Also assume local time zone when time without timezone information is passed to querying APIs.
Previously such a time was parsed in UTC timezone. Add `Z` to the end of the time string
if the old behaviour is preferred.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6721
2024-09-26 12:50:14 +02:00
Aliaksandr Valialkin
55ecf4f766
lib/logstorage: add blocks_count pipe
This pipe is useful for debugging purposes when the number of processed blocks must be calculated for the given query:

    <query> | blocks_count

This helps detecting the root cause of query performance slowdown in cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070
2024-09-25 19:18:38 +02:00
Dima Lazerka
465c7ad045
docs: fixes misspelled typos
Also tried to make it catch "Authorisation" in the future, fixed a lot
of other misspells along the way, but didn't make it catch
"Authorisation" anyway.

- Fix misspelled "Authorization" header name
- Fix misspelled "organization"
- Fix more misspells
2024-09-13 13:19:03 +02:00
Aliaksandr Valialkin
c5badeea08
lib/logstorage: substitute | operator with or operator at math pipe
This is needed for avoiding confusion between the `|` operator at `math` pipe and `|` pipe delimiter.
For example, the following query was parsed unexpectedly:

   * | math foo / bar | fields x

as

   * | math foo / (bar | fields) as x

Substituting `|` with `or` inside `math` pipe fixes this ambiguity.
2024-09-06 22:43:29 +02:00
Andrii Chubatiuk
77c3bbf3fc
docs: updated guides structure, removed deprecated sort option (#6767)
### Describe Your Changes

* `sort` param is unused by the current website engine, and was present only for compatibility
with previous website engine. It is time to remove it as it makes no effect
* re-structure guides content into folders to simplify assets management

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 35d77a3bed)
2024-08-07 16:59:22 +02:00
Aliaksandr Valialkin
a135a4dcfa
Revert "removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)"
This reverts commit 2e9b1efeb9.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 14:41:13 +02:00
Aliaksandr Valialkin
edc16e171e
Revert "fixed victorialogs relative links (#6693)"
This reverts commit 683a69d53d.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 14:40:31 +02:00
Artem Navoiev
1951555ffe
docs: fix link in logsql doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-25 14:38:11 +02:00
Andrii Chubatiuk
d25f88b912
fixed victorialogs relative links (#6693)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-25 13:41:19 +02:00
Andrii Chubatiuk
9a051fc80f
removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)
### Describe Your Changes

Use relative markdown references, removed `{{< ref >}}` shortcodes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-25 13:20:05 +02:00
Andrii Chubatiuk
6b97044d8a
view documentation locally (#6677)
- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-25 12:27:05 +02:00
Arkadii Yakovets
8645b2cc8e
docs: add spellcheck command (#6562)
### Describe Your Changes

Implement spellcheck command:
  - add cspell configuration files
  - dockerize spellchecking process
  - add Makefile targets

This PR adds a standalone `make spellcheck` target to check `docs/*.md` files for spelling
errors. The target process is dockerized to be run in a separate npm environment.

Some `docs/` typo fixes also included.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit fabf0b928e)
2024-07-11 12:40:24 +02:00
Aliaksandr Valialkin
76a58ae08d
lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:46:03 +02:00
Aliaksandr Valialkin
d0dca7b8c5
lib/logstorage: add delimiter between log chunks returned from | stream_context pipe 2024-07-01 01:46:02 +02:00
Aliaksandr Valialkin
76053a0ef0
docs/VictoriaLogs: typo fixes 2024-06-28 19:26:32 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
3eecc3de8c
docs/VictoriaLogs: typo fixes 2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
813a7aec6c
docs: return back spellcheck fixes from the commit 92b22581e6, which were accidentally reverted in 45cf83cd8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470
2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow (#6470)"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow (#6470)
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress 2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
c6b6900152
docs/VictoriaLogs/LogsQL.md: typo fix: replace from _json with from _msg where appropriate 2024-06-05 10:15:56 +02:00
Aliaksandr Valialkin
be4aefd133
docs/VictoriaLogs/LogsQL.md: substitute TAG with APP-NAME[PROCID] in rfc3164 syslog message format at unpack_syslog pipe docs 2024-06-05 10:13:58 +02:00
Aliaksandr Valialkin
a200fb433a
lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin
f8336783c8
docs/VictoriaLogs/LogsQL.md: typo fix: substitute "'" with "" in front of John` 2024-06-05 09:51:29 +02:00
Aliaksandr Valialkin
9e79a632c1
docs/VictoriaLogs/LogsQL.md: typo fix: its -> it 2024-06-05 09:46:35 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
Aliaksandr Valialkin
1a1ee87b46
docs/VictoriaLogs/LogsQL.md: mention unpack_syslog pipe in the Transformations section 2024-06-04 03:17:14 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
Arkadii Yakovets
a6655322b1
docs: fix docs/ and README.md spelling errors (#6362)
Fixes `docs/` and `README.md` typos and errors.

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>

(cherry picked from commit c740a8042e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:53:33 +02:00
Aliaksandr Valialkin
b994e64567
docs/VictoriaLogs: typo fixes 2024-05-30 16:35:25 +02:00
Aliaksandr Valialkin
540bbb63a2
lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
Aliaksandr Valialkin
e83fd4a117
lib/logstorage: work-in-progress 2024-05-29 01:52:34 +02:00
Aliaksandr Valialkin
6ddc618731
docs/VictoriaLogs: typo fixes 2024-05-28 22:48:57 +02:00
Aliaksandr Valialkin
79c03fc35f
lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin
afa597ce2a
lib/logstorage: work-in-progress 2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin
6427b3c3c0
lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
Aliaksandr Valialkin
03fe4c8963
lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress 2024-05-25 00:31:55 +02:00
Aliaksandr Valialkin
7a2a2f173e
lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00