Commit Graph

88 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
fc3d826d7f
all: add Windows build for VictoriaMetrics
This commit changes background merge algorithm, so it becomes compatible with Windows file semantics.

The previous algorithm for background merge:

1. Merge source parts into a destination part inside tmp directory.
2. Create a file in txn directory with instructions on how to atomically
   swap source parts with the destination part.
3. Perform instructions from the file.
4. Delete the file with instructions.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since the remaining files with instructions is replayed on the next restart,
after that the remaining contents of the tmp directory is deleted.

Unfortunately this algorithm doesn't work under Windows because
it disallows removing and moving files, which are in use.

So the new algorithm for background merge has been implemented:

1. Merge source parts into a destination part inside the partition directory itself.
   E.g. now the partition directory may contain both complete and incomplete parts.
2. Atomically update the parts.json file with the new list of parts after the merge,
   e.g. remove the source parts from the list and add the destination part to the list
   before storing it to parts.json file.
3. Remove the source parts from disk when they are no longer used.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since incomplete partitions from step 1 or old source parts from step 3 are removed
on the next startup by inspecting parts.json file.

This algorithm should work under Windows, since it doesn't remove or move files in use.
This algorithm has also the following benefits:

- It should work better for NFS.
- It fits object storage semantics.

The new algorithm changes data storage format, so it is impossible to downgrade
to the previous versions of VictoriaMetrics after upgrading to this algorithm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin
7d37ab63a5
Makefile: update golangci-lint from v1.51.1 to v1.51.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.51.2
2023-03-12 17:08:34 -07:00
Aliaksandr Valialkin
6149e4c319
Makefile: update golangci-lint from v1.50.1 to v1.51.1 2023-02-07 11:08:38 -08:00
Aliaksandr Valialkin
87a4943bea
Makefile: remove trailing space after golangci-lint run command
It is left after ec2c82e800
2023-01-05 18:09:05 -08:00
Luke Palmer
51d956225d
Lint and errcheck using golangci-lint (#3558) 2023-01-05 17:00:17 -08:00
Aliaksandr Valialkin
a6a2e35a83
Makefile: update golangci-lint version from v1.48.0 to v1.50.1 2022-12-20 13:10:42 -08:00
Aliaksandr Valialkin
2e714aa2d3
Makefile: publish release docker images at DockerHub with the stable tag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2911
2022-12-20 12:27:24 -08:00
Aliaksandr Valialkin
9d3c121feb
deployment/docker: use cluster-latest tag for docker images from cluster components
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3505
2022-12-20 10:38:51 -08:00
Zakhar Bessarab
de0e4f1f7e
make: make openssl output parsing symbol number agnostic (#3429) 2022-12-02 19:10:12 -08:00
Aliaksandr Valialkin
e7c2d057e6
Makefile: remove docs/*.tmp files after running sed command there 2022-10-06 15:25:27 +03:00
Aliaksandr Valialkin
9f1632b30b
Makefile: add missing "=" char between "-i" flag and its value for sed
This is a follow-up after 78af27f955
2022-10-06 15:10:52 +03:00
Roman Khavronenko
92f4d6a338
docs: when modifying docs in place allow storing backups (#3205)
The stored backups would help to identify docs corruption
but aren't needed for commiting. So `.tmp` backup files
are also git-ignored.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 15:10:52 +03:00
Roman Khavronenko
238401e3ed
docs: udpate Datadog section (#3190)
The `docs-sync` command was updated to modify images path
for assets in `docs/` folder. The change allows to refer images from `docs`
without copying them to the root folder.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-05 22:36:35 +03:00
Aliaksandr Valialkin
ae2f669e73
Makefile: run errcheck for all the app/... subdirs 2022-09-30 18:40:06 +03:00
Aliaksandr Valialkin
47ebf6e836
Makefile: fix -compat value passed to go mod tidy 2022-09-19 15:14:01 +03:00
Roman Khavronenko
d071e39694
bump Go version to 1.19.1 (#3108)
The reason is to cover vulnerability GO-2022-0969
Found in: net/http@go1.18.5
Fixed in: net/http@go1.19.1
More info: https://pkg.go.dev/vuln/GO-2022-0969

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 13:43:27 +03:00
Aliaksandr Valialkin
21d608b210
Makefile: remove github-create-release and github-upload-assets commands from publish-release
This is a follow-up for b9231c715a
2022-09-08 21:03:00 +03:00
Aliaksandr Valialkin
6cbe85a08a
Makefile: consistently use go install instead of go get for installing various binaries needed during build/test/check of the code
`go install` is the preferred way for installing go binaries starting
from the minimum supported Go version for VictoriaMetrics - Go1.18 -
see https://tip.golang.org/doc/go1.18#go-command
2022-09-08 18:46:08 +03:00
Aliaksandr Valialkin
c083286a92
Makefile: check for vulnerabilities in used Go packages with govulncheck when running make check-all
See https://go.dev/blog/vuln
2022-09-08 18:46:00 +03:00
Aliaksandr Valialkin
f6b3ad0c37
docs/Release-Guide.md: require manual push of the created release tags to public Github
The automated push of release tags to Github require specifying the remote repository name
when doing `git push <remote-repo-name> v1.xx.y`.
The remote repository name can differ in different environments,
so it cannot be put into Makefile rule.

TODO: create a Makefile rule, which generates standard remote names for public
and private repositories in Git, so `git push` for release tags could be automated then.
2022-09-08 15:00:34 +03:00
Aliaksandr Valialkin
6fe1f895f0
Makefile: properly push public tags 2022-09-02 23:19:03 +03:00
Aliaksandr Valialkin
0be941c99b
Makefile: push v1.xx.y and v1.xx.y-cluster tags to github before creating the v1.xx.y release at github
Otherwise Github creates the v1.xx.y tag on itself when creating the release
2022-09-02 21:49:10 +03:00
Max Golionko
0359fce052
simplify release process (#3012)
* simplify release process

* address comments

* address comments

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-31 02:28:45 +03:00
Aliaksandr Valialkin
40c2fae617
vendor: make vendor-update 2022-08-15 00:54:50 +03:00
Aliaksandr Valialkin
bfa490e921
Makefile: update golangci-lint from v1.47.1 to v1.48.0
This is needed for adding support for Go 1.19
2022-08-07 22:33:40 +03:00
Aliaksandr Valialkin
573f6c8d6c
Makefile: remove redundant -mod=vendor option when running Go tools
The `-mod=vendor` is automatically set when there is a `vendor` directory
starting from Go1.14 - see https://go.dev/doc/go1.14#go-command

Since the minimum supported Go version for VictoriaMetrics is Go1.17,
then the `-mod=vendor` option is no longer needed.
2022-08-07 20:40:41 +03:00
Denys Holius
9b4024cd62
deployment/docker/Makefile: added docker-scan (#2916)
* deployment/docker/Makefile: added docker-scan

docker-scan based on native 'docker scan' function that use snyk.io, see https://docs.docker.com/engine/scan/

* set to call 'docker-scan after release binaries but before publishing
2022-08-02 13:26:43 +03:00
Denys Holius
d635169c90
Update golangci version to latest v1.47.1 (#2890)
See https://github.com/golangci/golangci-lint/releases/tag/v1.47.1
2022-07-19 19:31:16 +03:00
Aliaksandr Valialkin
da6c85a2f6
all: follow-up for d99ba3481b 2022-07-13 17:17:08 +03:00
Dmytro Kozlov
4e4def9df8
Rename release packages (#2810)
* makefile: add os to each release file

* makefile: update vmutils arm64

* makefile: update victoria-metrics release process

* makefile: update publish with os

* makefile: update publish with os

* makefile: change tar library

* update release logic

* copy all releases

* sort command by GOOS

* rollback commands

* rollback OSARCH

* fix commands

* cleanup

* fix windows build

* sort build by GOOS, update README.md
2022-07-13 17:11:01 +03:00
Dmytro Kozlov
6add79143b
removed redundant return (fixed linter) (#2647)
* removed redundant return

* updated lint package version
2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin
d49dbec662
Makefile: explicitly specify go1.17 compatibility when running go mod tidy at make vendor-update
This is needed because go1.17 is the minimum supported version of Go,
which is needed for building VictoriaMetrics
2022-05-20 14:42:06 +03:00
Denys Holius
b0fc218dc9
Update golangci version to latest v1.46.1 (#2579) 2022-05-13 16:59:16 +03:00
Denys Holius
8752ed9985
Update golangci version to latest v1.46.0 (#2560)
Update golangci version to latest https://github.com/golangci/golangci-lint/tree/v1.46.0
2022-05-09 17:58:37 +03:00
Denys Holius
259ea7ee39
Update golangci version to latest v1.45.1 (#2360) 2022-03-24 19:18:02 +02:00
Denys Holius
bf0b31e7bb
Bump golangci version (#2214) 2022-02-18 12:26:35 +02:00
Denys Holius
588f0ab4ee
Update version of golangci-lint
Update version of golangci-lint from v1.43.0 to v1.44.0 in Makefile.
See https://github.com/golangci/golangci-lint/releases/tag/v1.44.0 .
2022-01-27 13:22:22 +02:00
Aliaksandr Valialkin
7cbb81c2a6
Makefile: add TAG=v... make publish-release rule for building and publishing a release for the given TAG 2021-11-08 12:29:24 +02:00
Denys Holius
2d1b24217a
bumped golangci-lint to the latest 1.43 (#1781) 2021-11-04 12:16:55 +02:00
Aliaksandr Valialkin
6bc10f0623
lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed
This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels.
See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets

This is a follow-up after 9882cda8b9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728
2021-10-22 13:55:39 +03:00
Roman Khavronenko
b42f7fc185 Docs review (#1330)
* re-order components by prioritizing Cluster-VictoriaMetrics.md

* drop Home.md since it just duplicates other links
2021-05-26 12:32:59 +03:00
Aliaksandr Valialkin
69e365cd48 Makefile: update golangci-lint from v1.29.0 to v1.40.1 2021-05-20 18:30:24 +03:00
Aliaksandr Valialkin
36a79b7fd3 Add make check-licenses rule for the ability to manually check licenses in vendored dependencies
This is a follow-up for c687536956
2021-05-10 12:01:37 +03:00
Aliaksandr Valialkin
7f573a9564 docs/vmbackupmanager.md: sync with app/vmbackupmanager/README.md 2021-04-22 14:45:31 +03:00
Lapo Luchini
19906dc30e Allow specifying build date from a variable (#1200)
Just like the existing infrastructure for `BUILDINFO_TAG`, this can ease the production of [reproducible builds](https://wiki.freebsd.org/ReproducibleBuilds).
(e.g. in FreeBSD the date the port was committed is used at build time, not the actual build time, so that an identical port produced at different times produces an identical executable)
2021-04-10 15:47:53 +03:00
Aliaksandr Valialkin
18cad589d7 docs/Cluster-VictoriaMetrics.md: follow-up after c6a8ebb11f 2021-04-07 13:46:56 +03:00
Roman Khavronenko
ff3711eea2 docs: update docs ordering and formatting (#1192)
The major change is adding `sort` directive to docs. For those docs which are copied
from internal packages `sort` is added via makefile command. For the rest it is added
manually since they're updated manually as well.

The rest of changes is connected with markdown formatting. For example, changing headers
in some files (`##` => `#`) makes navigation on .github.io to look better. This especially
useful for `changelog` docs.

Table of contents for `vmctl` is dropped, since we already have it autogenerated on .github.io.

No link changes expected. The corresponding PR to `cluster` branch will be made in follow-up PR.
2021-04-07 13:43:01 +03:00
Aliaksandr Valialkin
2585058a5f Makefile: prepare arm64 and amd64 release archives for cluster version on make release command 2021-04-05 23:01:45 +03:00
Aliaksandr Valialkin
5edec0e57e app/vmgateway: publish docs 2021-04-02 23:10:10 +03:00
Aliaksandr Valialkin
3caac5edd4 Makefile: prepare vmutils-windows-*.zip archive on make release-vmutils command
The archive contains the following executables for Windows:

* vmagent
* vmalert
* vmauth
* vmctl

Other components - vmbackup, vmrestore, victoria-metrics - aren't supported for Windows yet
2021-03-16 20:54:10 +02:00