Commit Graph

129 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
5a9abe9783
Makefile: upgrade go mod tidy compatibility mode from Go1.22 to Go1.23
This is a follow-up for 1b9f3b39b4
2024-09-08 21:05:22 +02:00
Aliaksandr Valialkin
69171b3075
Makefile: remove '-d' flag from 'go get', since this flag is deprecated
See https://go.dev/doc/go-get-install-deprecation
2024-09-08 20:11:24 +02:00
Aliaksandr Valialkin
d04dbbc7b5
Makefile: update golangci-lint from v1.60.1 to v1.60.3
See https://github.com/golangci/golangci-lint/releases/tag/v1.60.3
2024-09-05 16:57:47 +02:00
rtm0
602bedf362
testing: allow disabling fsync to make tests run faster ()
### Describe Your Changes

fsync() ensures that the data is written to disk. In production this is
needed for data durability. However, during the development, when the
unit tests are run, this level of durability is not needed. Therefore
fsync() can be disabled which will makes test runs two times faster.

The disabling is done by setting the `DISABLE_FSYNC_FOR_TESTING`
environment variable. The valid values for this variable are the same as
the values of the arg of `go doc strconv.ParseBool`:

```
1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False.
```

Any other value means `false`.

The variable is set for all test build targets. Compare running times:

Build Target | DISABLE_FSYNC_FOR_TESTING=0 | DISABLE_FSYNC_FOR_TESTING=1
----------------- | ------------------------------------------------ |
-------------------------------------------------
make test | 1m5s  | 0m22s
make test-race | 3m1s | 1m42s
make test-pure | 1m7s | 0m20s
make test-full | 1m21s | 0m32s
make test-full-386 | 1m42s | 0m36s

When running tests for a given package, fsync can be disabled as
follows:

```shell
DISABLE_FSYNC_FOR_TESTING=1 go test ./lib/storage
```

Disabling fsync() is intended for testing purposes only and the name of
the variables reflects that.

What could also have been done but haven't:

- lib/filestream/filestream.go: `Writer.MustFlush()` also uses f.Sync()
but nothing has been done to it, because the Writer.MustFlush() is not
used anywhere in the VM codebase. A side question: what is the general
policy for the unused code?
- lib/filestream/filestream.go: Writer.Write() calls `adviceDontNeed()`
which calls unix.Fdatasync(). Disabling it could potentially improve
running time, but running tests with this code disabled has shown
otherwise.

### Checklist

The following checks are **mandatory**:

- [ x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>

(cherry picked from commit 334cd92a6c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-30 11:18:21 +02:00
Zakhar Bessarab
4cda699972
make: reset permissions when creating release archive ()
### Describe Your Changes

Forcefully set permissions for release binaries to 1000:1000. This helps
to avoid issues such as
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6788 on
systems with limitations around UID:GID configuration.

"1000" UID and GID is widely used by linux distributions for the first
user in the system.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 654c1cb9d1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-21 14:14:33 +02:00
hagen1778
d4c334b705
Makefile: update golangci-lint from v1.59.1 to v1.60.1
See https://github.com/golangci/golangci-lint/releases/tag/v1.60.1

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 220b1659b6)
2024-08-19 17:45:42 +02:00
Andrii Chubatiuk
56a6e680e3
docs: grouped changelog docs, removed old make commands, replaced docs in root README with official docs links ()
### Describe Your Changes

- replace docs in root README with a link to official documentation
- remove old make commands for documentation
- remove redundant "VictoriaMetrics" from document titles
- merge changelog docs into a section
- rm content of Single-server-VictoriaMetrics.md as it can be included from docs/README
- add basic information to README in the root folder, so it will be useful for github users
- rm `picture` tag from docs/README as it was needed for github only, we don't display VM logo at docs.victoriametrics.com
- update `## documentation` section in docs/README to reflect the changes
- rename DD pictures, as they now belong to docs/README

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 58e667c895)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-31 16:15:08 +02:00
Andrii Chubatiuk
6b97044d8a
view documentation locally ()
- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-25 12:27:05 +02:00
Arkadii Yakovets
8645b2cc8e
docs: add spellcheck command ()
### Describe Your Changes

Implement spellcheck command:
  - add cspell configuration files
  - dockerize spellchecking process
  - add Makefile targets

This PR adds a standalone `make spellcheck` target to check `docs/*.md` files for spelling
errors. The target process is dockerized to be run in a separate npm environment.

Some `docs/` typo fixes also included.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit fabf0b928e)
2024-07-11 12:40:24 +02:00
hagen1778
db9a1bb59e
docs: follow-up after c341369fc1
* account for `source` tag in `docs-sync` command
* run `make docs-sync`

c341369fc1
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1c155a0cd1)
2024-07-10 12:26:26 +02:00
Aliaksandr Valialkin
3c59e424c4
Makefile: add clean-checkers rule for dropping checker apps used in make check-all
The `make clean-checkers` command may be useful when the locally installed checker apps are too old,
so they must be updated to the newly requested versions by `make check-all`. In this case the fix is to run

    make clean-checkers check-all
2024-06-25 20:30:25 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow ()"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow ()
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Aliaksandr Valialkin
b74dbdbc82
Makefile: update golangci-lint from v1.57.1 to v1.59.1
See https://github.com/golangci/golangci-lint/releases/tag/v1.59.1

Thanks to ccoVeille for the suggestion at https://www.reddit.com/r/golang/comments/1dcl4ne/log_viewer_in_go/l82vsfd/

(cherry picked from commit 239a7b6e6f)
2024-06-11 17:05:12 +02:00
Aliaksandr Valialkin
817caae3f6
go.mod: update the required Go version from 1.21 to 1.22
This is a follow-up for 95222b2079
2024-05-12 16:08:39 +02:00
Denys Holius
7a5f3e8ef9
Makefile: bump version of golangci-lint to the latest v1.57.1 ()
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 0c0ed61ce7)
2024-03-25 15:42:52 +01:00
Aliaksandr Valialkin
4e5cbc5dc6
deployment/docker: downgrade Go builder from 1.22.0 to 1.21.7
Go1.22.0 contains the bug https://github.com/golang/go/issues/65705 ,
which prevents vmagent from normal operation.
2024-02-29 13:52:58 +02:00
Aliaksandr Valialkin
0b2087b6e5
Makefile: run go mod tidy with -compat=1.22 after 95222b2079 2024-02-12 22:36:32 +02:00
Alexander Marshalov
1f858eb417
Follow up after https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5685 () 2024-01-26 20:08:15 +01:00
Alexander Marshalov
711ace0582
Make a makefile work on a MacOS and Linux () 2024-01-26 20:07:20 +01:00
Aliaksandr Valialkin
db6dadf1f7
docs: convert png images to webp in all the docs except of docs/operator/*
This reduces the size of docs/* folder from 33MB to 18MB

Images inside docs/operator/* must be converted at the https://github.com/VictoriaMetrics/operator/tree/master/docs
and then the updated images must be automatically propagated to the docs/operator/*

This is a follow-up for d3f919df3e

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5206
2023-11-22 19:29:47 +02:00
Aliaksandr Valialkin
c285fca256
Makefile: allow specifying the needed concurrency for make via MAKE_CONCURRENCY env var 2023-11-21 01:11:36 +02:00
Aliaksandr Valialkin
6b75523468
Makefile: speedup release, publish and crossbuild rules by using parallel make 2023-11-20 23:07:11 +02:00
Aliaksandr Valialkin
00f8c55efd
Makefile: remove package-base dependency from publish rule, since this dep is set inside all the publish-* dependencies
This is a follow-up for d4099a75be
2023-11-15 12:58:30 +01:00
Aliaksandr Valialkin
996e746c2c
Makefile: update golangci-lint version from v1.54.2 to v1.55.1
See https://github.com/golangci/golangci-lint/releases/tag/v1.55.1
2023-11-02 21:42:35 +01:00
Haleygo
130e0ea5f0
vmalert-tool: implement unittest ()
1. split package rule under /app/vmalert, expose needed objects
2. add vmalert-tool with unittest subcmd

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
2023-10-16 14:12:06 +02:00
Aliaksandr Valialkin
0bc0d2610c
go.mod: increase the minimum supported Go version from Go1.19 to Go1.20 2023-09-07 12:18:12 +02:00
Aliaksandr Valialkin
d8afd7fe98
Makefile: update golangci-lint from v1.51.2 to v1.54.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.54.2
2023-09-01 10:25:49 +02:00
Aliaksandr Valialkin
0609d8c8c5
Makefile: cleanup bin/ directory at the beginning of make publish-release command
This is needed in order to prevent from non-build artifacts to be uploaded to Github release page
2023-08-29 13:16:20 +02:00
Roman Khavronenko
c81e90223c
dashboards: provide copies of Grafana dashboards alternated with Vict… ()
dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-29 11:20:16 +02:00
Aliaksandr Valialkin
3e5e55d235
Revert "make: add goimports task ()"
This reverts commit 20b18e9feb.

Reason for revert: running goimports on `make check-all` introduces the following issues:

- It runs only on modified files, which weren't commited yet into git repository.
  This means the formatting for the remaining files becomes different comparing to the formatting
  for the changed files. This also means that the goimports has no any effect
  at github actions and when the changed code is already commited to git repository.
- `gomiports` performs formatting in the same way as gofmt, so `make fmt` becomes unnecessary.
  But when `gofmt` is substituted with `goimports`, then it performs unnecessary formatting for *.qtpl.go files.
  It is possible to make a hack, which will prepare a list of all the *.go files at lib/ and app/
  without the *.qtpl.go files, and then feed this list to `goimports`, but this looks too fragile
  for the task of just fixing the ordering of Go imports.

So it is better to leave source code formatting as is with `gofmt`, while manually fixing improper ordering
of Go import from time to time in dedicated commits until better solution arises.
2023-07-13 12:18:02 -07:00
Zakhar Bessarab
64149782ac
make: add goimports task ()
* make: add goimports task

Adds task to fix imports formatting implace.
Formats imports into:
- native library
- external libraries
- local packages based on github.com/VictoriaMetrics/VictoriaMetrics prefix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: add goimports install task

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: run goimports only for changed files

Applying goimports to all existing files would create a lot of problems with cherry-picking changes between different branches used for development. To avoid this it was decided to only run goimports on changed files to fix formatting gradually.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: update goimports to run on all changed files

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-13 12:17:31 -07:00
Denys Holius
ef29a41824
docs: use printf for Makefile:copy-docs section ()
printf handles new line char for cross-platform use
2023-07-06 22:20:53 -07:00
Max Golionko
75d9914916
CI: disable docker scan, enable auto release to sandbox ()
* disable docker scan

* disable nightly, enable auto release to sandbox

* remove whitespace
2023-07-06 22:02:48 -07:00
Artem Navoiev
258575441e
revert version of golanci
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:47:35 -07:00
Artem Navoiev
28301d1b41
remove deleted repo from the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:47:07 -07:00
Aliaksandr Valialkin
4f6eb775df
Makefile: remove trailing whitespace from copy-docs comments 2023-07-06 11:01:39 -07:00
Artem Navoiev
8dd2a263b9
desribe old link param
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit bf4711ecba)
2023-06-02 13:25:49 +02:00
Artem Navoiev
5e6fe50c53
return information about cluster
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

(cherry picked from commit 8f09569cb8)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:25:48 +02:00
Artem Navoiev
3185b8048c
update docs-sync Makefile command, add hugo front-matter
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

(cherry picked from commit f791811b15)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:24:41 +02:00
Aliaksandr Valialkin
ef4e5e4fc7
Makefile: add missing vmcluster-windows-amd64 target
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-25 15:34:21 -07:00
Aliaksandr Valialkin
fc3d826d7f
all: add Windows build for VictoriaMetrics
This commit changes background merge algorithm, so it becomes compatible with Windows file semantics.

The previous algorithm for background merge:

1. Merge source parts into a destination part inside tmp directory.
2. Create a file in txn directory with instructions on how to atomically
   swap source parts with the destination part.
3. Perform instructions from the file.
4. Delete the file with instructions.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since the remaining files with instructions is replayed on the next restart,
after that the remaining contents of the tmp directory is deleted.

Unfortunately this algorithm doesn't work under Windows because
it disallows removing and moving files, which are in use.

So the new algorithm for background merge has been implemented:

1. Merge source parts into a destination part inside the partition directory itself.
   E.g. now the partition directory may contain both complete and incomplete parts.
2. Atomically update the parts.json file with the new list of parts after the merge,
   e.g. remove the source parts from the list and add the destination part to the list
   before storing it to parts.json file.
3. Remove the source parts from disk when they are no longer used.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since incomplete partitions from step 1 or old source parts from step 3 are removed
on the next startup by inspecting parts.json file.

This algorithm should work under Windows, since it doesn't remove or move files in use.
This algorithm has also the following benefits:

- It should work better for NFS.
- It fits object storage semantics.

The new algorithm changes data storage format, so it is impossible to downgrade
to the previous versions of VictoriaMetrics after upgrading to this algorithm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin
7d37ab63a5
Makefile: update golangci-lint from v1.51.1 to v1.51.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.51.2
2023-03-12 17:08:34 -07:00
Aliaksandr Valialkin
6149e4c319
Makefile: update golangci-lint from v1.50.1 to v1.51.1 2023-02-07 11:08:38 -08:00
Aliaksandr Valialkin
87a4943bea
Makefile: remove trailing space after golangci-lint run command
It is left after ec2c82e800
2023-01-05 18:09:05 -08:00
Luke Palmer
51d956225d
Lint and errcheck using golangci-lint () 2023-01-05 17:00:17 -08:00
Aliaksandr Valialkin
a6a2e35a83
Makefile: update golangci-lint version from v1.48.0 to v1.50.1 2022-12-20 13:10:42 -08:00
Aliaksandr Valialkin
2e714aa2d3
Makefile: publish release docker images at DockerHub with the stable tag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2911
2022-12-20 12:27:24 -08:00
Aliaksandr Valialkin
9d3c121feb
deployment/docker: use cluster-latest tag for docker images from cluster components
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3505
2022-12-20 10:38:51 -08:00
Zakhar Bessarab
de0e4f1f7e
make: make openssl output parsing symbol number agnostic () 2022-12-02 19:10:12 -08:00