Aliaksandr Valialkin
894e5d2b9b
docs/vmbackup.md: add a link to vmbackuper tool
2020-05-13 22:54:22 +03:00
Roman Khavronenko
415b1ddfb5
vmalert: check if remoteRead object was initied before calling Restore ( #473 )
...
The check for non-nil remoteRead was mistakenly dropped
during refactoring which caused panics when `vmalert`
wasn't configured with `remoteRead` flag.
2020-05-13 19:32:58 +01:00
Roman Khavronenko
db7dd96346
vmalert: fix flag names and description in README ( #475 )
...
Change also adds the recommendation for `remotewrite`
queue error.
2020-05-13 19:32:21 +01:00
肖贝贝
ba48438b06
Feat/vmalert add max queue size ( #472 )
...
* feat: add remoteWrite.maxQueueSize to reduce queue full
* rename remote(write|read) flags to remote(Write|Read) for the sake of consistency
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
2020-05-13 18:58:56 +01:00
Aliaksandr Valialkin
7882a0dbbf
app/vmselect/promql: suppress "SA4006: this value of dstValues
is never used" error in golangci-lint
2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin
96e001d254
app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig
args are set
...
This bug could result in incorrect relabeling and metrics' drop.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467
2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin
faf92a0965
app/vmselect/promql: fix any(..)
calculations - return all the data points instead of the first one
2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin
cc311e20fe
app/vmselect/promql: add any(x) by (y)
aggregate function, which returns any time series from q
for each group y
2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin
574289c3fb
app/vmselect/promql: support for sum(x) by (y) limit N
syntax in order to limit the number of output time series after aggregation
2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin
0a134ace63
app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin
8300cc17af
app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set
2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin
6273385618
app/vmagent/remotewrite: properly dial TCP6 addresses set via -remoteWrite.url
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469
2020-05-12 15:22:29 +03:00
Aliaksandr Valialkin
dbd0c552d5
lib/storage: gradually pre-populate per-day inverted index for the next day
...
This should prevent from CPU usage spikes at 00:00 UTC every day when
inverted index for new day must be quickly created for all the active time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430
2020-05-12 12:13:05 +03:00
Roman Khavronenko
8c8ff5d0cb
vmalert: cleanup and restructure of code to improve maintainability ( #471 )
...
The change introduces new entity `manager` which replaces
`watchdog`, decouples requestHandler and groups. Manager
supposed to control life cycle of groups, rules and
config reloads.
Groups export an ID method which returns a hash
from filename and group name. ID supposed to be unique
identifier across all loaded groups.
Some tests were added to improve coverage.
Bug with wrong annotation value if $value is used in
templates after metrics being restored fixed.
Notifier interface was extended to accept context.
New set of metrics was introduced for config reload.
2020-05-10 17:58:17 +01:00
Nikolay Khramchikhin
9e8733ff65
vmalert config reload
...
added config hot reload for vmalert with sighup and api call
2020-05-09 10:32:12 +01:00
Aliaksandr Valialkin
baedb25936
docs/vmauth.md: fix a link to docker images
2020-05-08 14:10:04 +03:00
Aliaksandr Valialkin
51291015a5
app/vmagent: return 200 from /-/reload
endpoint as Prometheus does
2020-05-07 19:30:30 +03:00
Aliaksandr Valialkin
6afb25fd08
docs/{vmagent,vmauth}: small clarifications in the docs
2020-05-07 12:55:20 +03:00
Aliaksandr Valialkin
653d51694a
app/vmauth: prevent from attacks with ..
in path for accessing resources outside the configured url_prefix
2020-05-07 12:55:18 +03:00
Aliaksandr Valialkin
8a00807f60
app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url
2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin
b69eb7bf38
app/vmagent: properly set client-side TLS certificates for -remoteWrite.url
. Previously they were mistakenly set as server-side
2020-05-06 16:50:30 +03:00
Aliaksandr Valialkin
e8936c9cb3
docs/vmagent.md: small fixes
2020-05-06 14:49:18 +03:00
Aliaksandr Valialkin
3f52a97f9b
lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs
2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin
08320cfcf4
docs/{vmauth,vmagent}: fix ports for profiling
2020-05-05 20:15:47 +03:00
Aliaksandr Valialkin
f65930b34d
docs/vmauth.md: mention that we can help creating customized proxy
2020-05-05 12:34:42 +03:00
Aliaksandr Valialkin
266327642b
docs/{vmagent,vmauth}: add Profiling
section
2020-05-05 11:45:13 +03:00
Aliaksandr Valialkin
0c7cddfca6
docs: add vmauth.md
2020-05-05 11:17:23 +03:00
Aliaksandr Valialkin
e767aedd17
app/vmauth: add initial version of vmauth. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details
2020-05-05 10:54:17 +03:00
Aliaksandr Valialkin
b5a780930d
docs/vmagent.md: /targets
page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead
2020-05-05 10:54:14 +03:00
Roman Khavronenko
0ba1b5c71b
app/vmalert: restore alerts state from datasource metrics ( #461 )
...
* app/vmalert: restore alerts state from datasource metrics
Vmalert will restore alerts state for rules that have `rule.For` > 0 from previously written timeseries via `remotewrite.url` flag.
* app/vmalert: mention remotewerite and remoteread configuration in README
2020-05-05 00:51:22 +03:00
Aliaksandr Valialkin
40c3ffb359
lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin
432187ac3b
app/vminsert: add /-/reload
handler in the same way as for vmagent
2020-04-30 02:15:39 +03:00
DexterZhang
67511d4165
feat(vmagent): add promscrap config reload suppport via http ( #450 )
...
* feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload`
* fix: typo fix
2020-04-30 02:00:32 +03:00
Aliaksandr Valialkin
43c39dc36c
vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp
...
The upstream fasthttp may contain issues like 996610f021
,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
2020-04-29 17:33:34 +03:00
Artem Navoiev
4487b454a8
Update README.md
2020-04-29 12:39:15 +03:00
Aliaksandr Valialkin
57407cca83
app/vmselect/promql: remove -search.maxPointsPerTimeseries
command-line flag
...
Limit the estimated time series count after aggregation with grouping by the number of source time series.
2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin
4e4f57b121
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
17d96e4503
app/vmselect: add -search.estimatedSeriesCountAfterAggregation
command-line flag for tuning the probability of OOMs vs false-positive not enough memory
errors
2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin
1397612117
app/vmalert: added missing comments for public entities
2020-04-28 11:21:07 +03:00
Roman Khavronenko
3bfa41a95c
app/vmalert: initial remote-write support for alerts state persistence. ( #442 )
...
* app/vmalert: initial remote-write support for alerts state persistence.
If `remotewrite.url` flag is set, vmalert will send alerts state via remote-write protocol to remote storage. The sending is asynchronous to avoid blocking calls in rules evaluation loop.
* app/vmalert: merge with master
* app/vmalert: write both `instant` and `for` alerts timeseries states in remote storage.
2020-04-28 00:18:02 +03:00
Aliaksandr Valialkin
90670cb55e
app/vmalert: include it into the next release
2020-04-28 00:10:12 +03:00
Aliaksandr Valialkin
b768bc9a6a
lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs
2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin
b4afe562c1
lib/storage: postpone reading data from blocks during search
...
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.
This improves single-node VM performance on heavy queries by up to 2x.
2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
0224071ebe
lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config
2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin
fcf57f9883
app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks
...
This should minimize the number of disk seeks when reading data from temporary file.
2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin
6954d0edb7
lib/promscrape/discovery/gce: allow empty zone
arg in gce_sd_config
- in this case zones for the given project are automatically discovered
2020-04-26 14:34:11 +03:00
kreedom
2c18548e08
alert - rename validate function and flags ( #440 )
...
* alert - rename validate function and flags
2020-04-26 14:15:04 +03:00
kreedom
5f61d43db9
vmalert - validate template in labels ( #439 )
2020-04-26 13:53:57 +03:00
肖贝贝
eeadfccdc5
fix: fix vmalert template label not complete bug ( #435 )
...
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
2020-04-26 13:30:10 +03:00
Aliaksandr Valialkin
1f3fd93b58
docs/{vmbackup,vmrestore}.md: update -help
output
2020-04-24 22:44:21 +03:00
Jason Gardner
66af7e40f3
app/vmbackup: added ability to create and delete snapshots during backup ( #428 )
...
* app/vmbackup: added ability to create and delete snapshots during backup
Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422
* Add snapshot create and delete url flags
* Fixed errcheck warnings in build
2020-04-24 22:35:03 +03:00
Aliaksandr Valialkin
a596aec82c
app/vmselect: fix description for -search.resetCacheAuthKey
2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin
9ef5935552
lib/promscrape: initial implementation for gce_sd_configs
aga Prometheus-compatible service discovery for Google Compute Engine
2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin
364db13c9c
app/vmselect: add /api/v1/status/tsdb
page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
9ebc937685
app/vmselect: add -search.minStalenessInterval
command-line flag for removing gaps on graphs built from time series with irregular duration between samples
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin
fe57d46687
app/vmselect: merge -search.maxLookback
and -search.maxStalenessInterval
flags, since it has been appeared they have identical purpose :(
...
Leave both flags for backwards compatibility reasons.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin
851946af1e
deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var
...
For example, the following command will build VictoriaMetrics docker image on top of alpine image:
ROOT_IMAGE=alpine make package-victoria-metrics
2020-04-20 01:16:57 +03:00
Aliaksandr Valialkin
936fb0eac3
app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection
...
This should fix the following error when sending data to remote storage:
couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
2020-04-17 15:52:42 +03:00
Aliaksandr Valialkin
79fb595732
docs/vmagent.md: typo fix: unvailable -> unavailable
2020-04-17 13:11:31 +03:00
Aliaksandr Valialkin
546d26523c
app/vmagent/README.md: mention about prodmscrape.suppressScrapeErrors
2020-04-17 13:08:21 +03:00
Aliaksandr Valialkin
f41e6a7bd9
app/vmselect: properly apply -search.maxLookback
to queries sent to /api/v1/query
2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin
071fdf5518
lib/logger: add WARN level for logging expected errors such as invalid user queries
2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin
6f7f64f757
app/vmselect: handle timestamp(metric offset X)
the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin
067c7afebc
lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets
page
...
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin
0b76c27fa1
docs/vmagent.md: mention that vmagent supports kubernetes_sd_configs
now
2020-04-13 21:06:36 +03:00
hagen1778
2eed6c393f
vmalert: prepare package for external usage
...
* update README according to changes
* add Makefile with basic commands
2020-04-12 15:32:42 +03:00
kreedom
948f8b6b5f
[vmalert] fix linter issues
2020-04-12 15:08:11 +03:00
kreedom
8fca5f2819
[vmalert] add tests to webserver ( #413 )
2020-04-12 14:51:03 +03:00
Roman Khavronenko
7c9405f53d
Vmalert metrics ( #412 )
...
vmalert: add basic list of metrics
2020-04-11 20:42:01 +01:00
Roman Khavronenko
9f8cc8ae1b
Extend web responses for alerts: ( #411 )
...
vmalert: Extend web responses for alerts
* populate apiAlert object with additional fields
* return all active alerts, not only firing
* sort list of API alerts for deterministic output
* add helper for available path list
2020-04-11 16:49:23 +01:00
kreedom
90de3086b3
[vmalert] add webserver ( #410 )
...
* [vmalert] add webserver
2020-04-11 12:40:24 +03:00
Aliaksandr Valialkin
f94a090020
docs: update minimum supported Go version from 1.12 to 1.13
2020-04-07 13:38:37 +03:00
Roman Khavronenko
b099d84271
Vmalert/rules eval ( #400 )
...
* Initial rules evaluation support.
Rules are now store alerts state in private field `alerts`. Every evaluation updates
the alerts and state. Every unique metric received from datastore represents a unique alert,
uniqueness is guaranteed by hashing ordered labelset.
* merge with master
* cleanup
* support endAt parameter as 3*evaluationInterval for active alerts
* make golint happy
2020-04-06 14:44:03 +03:00
Aliaksandr Valialkin
895cadfae7
app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request
...
This header is required by Cortex (and, probably, other remote storage systems).
See 9c1f44d090/docs/apis.md (remote-api)
.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399
2020-04-04 16:24:56 +03:00
Aliaksandr Valialkin
57704aa584
app/victoria-metrics: add -selfScrapeInstance
and -selfScrapeJob
flags for tuning labels for self-scraped metrics
2020-04-04 14:57:22 +03:00
Aliaksandr Valialkin
f9b24d4899
app/vmselect/promql: keep metric name after applying first_over_time
and last_over_time
functions
2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin
b38d048dd9
app/vmstorage: add vm_free_disk_space_bytes
metric for monitoring the remaining disk space at -storageDataPath
2020-04-01 23:08:58 +03:00
kreedom
298eb0a0f8
[vmalert] improve external url handling
2020-04-01 22:29:11 +03:00
kreedom
12fe915b48
[vmalert] add prometheus template function ( #396 )
...
* [vmalert] add prometheus template function
* make linter be happy
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-04-01 18:17:53 +03:00
Aliaksandr Valialkin
e71519b8b2
app/victoria-metrics/testdata: add a test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
2020-03-31 12:51:25 +03:00
Aliaksandr Valialkin
318326c309
lib/storage: properly handle {label=~"foo|"}
filters as Prometheus does
...
Such filters must match all the time series with `label="foo"` plus all the time series without `label`
Previously only time series with `label="foo"` were matched.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
0c485f14d1
app/vmselect/prometheus: allow passing relative time to start
, end
and time
args of /api/v1/*
queries
2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin
2ebf7d86ff
app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()
2020-03-29 21:50:28 +03:00
kreedom
bf6c24d0f4
[vmalert] config parser ( #393 )
...
* [vmalert] config parser
* make linter be happy
* fix test
* fix sprintf add test for rule validation
2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin
bd156cd088
docs/vmagent.md: add prometheus remote_write proxy
use case
2020-03-28 23:16:38 +02:00
Dmitry Naumov
27373807c1
Rootless docker images by default ( #358 )
...
* Rootless docker images by default
* Migrate to rootless base image
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-03-27 21:23:50 +02:00
kreedom
b22da547a2
[vmalert] - parse template annotaions ( #387 )
...
* [vmalert] - parse template annotations
2020-03-27 18:31:16 +02:00
Aliaksandr Valialkin
3c7c71a49c
app/vmselect: adjust label_map()
handling for corner cases
...
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series
All the unmatched labels are kept unchanged.
2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin
4fc4912f0c
app/vmalert/datasource: typo fix in docs: Labels -> Label
2020-03-13 12:22:33 +02:00
kreedom
a746cb62b6
vmalert add vm datasource, change alertmanager ( #364 )
...
* vmalert add vm datasource, change alertmanager
* make linter be happy
* make linter be happy.2
* PR comments
* PR comments.1
2020-03-13 12:19:31 +02:00
Aliaksandr Valialkin
fdc2a9d1d7
app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN)
function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin
8023ad7dbd
app/vmselect: add -search.maxStalenessInterval
for tuning Prometheus data model closer to Influx-style data model
2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin
fcd91795d5
app/vmagent: mention that vmagent can filter data
2020-03-11 16:22:39 +02:00
Aliaksandr Valialkin
301c2acd61
app/vmstorage: return 500 status code instead of 200 status code on internal errors inside /snapshot/*
handlers
2020-03-10 23:51:55 +02:00
Aliaksandr Valialkin
e17702fada
app/vmselect: add optional max_rows_per_line
query arg to /api/v1/export
...
This arg allows limiting the number of data points that may be exported on a single line.
2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin
1fe66fb3cc
app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv
2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin
5fe38a84eb
app/vmagent: properly apply -remoteWrite.sendTimeout
to fasthttp.HostClient
2020-03-09 13:31:55 +02:00
Aliaksandr Valialkin
c386c5de57
app/vmagent: properly add labels set via -remoteWrite.label
to metrics before sending them to -remoteWrite.url
2020-03-06 19:26:58 +02:00
Aliaksandr Valialkin
c5f894b361
Makefile: add build and test rules with enabled race detector. These rules have -race
suffix
...
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin
754db0d22e
app/vmagent/README.md: small fixes
2020-03-04 18:14:47 +02:00
Aliaksandr Valialkin
772312bf7b
app/vmagent/README.md: typo fix
2020-03-04 18:05:09 +02:00
Aliaksandr Valialkin
871abfab7a
app/vmagent/README.md: clarification
2020-03-04 18:03:48 +02:00
Aliaksandr Valialkin
007c591de8
app/vmagent/README.md: add iot and edge monitoring
use case
2020-03-04 18:01:34 +02:00
Aliaksandr Valialkin
474a09c0f1
app/vmagent/README.md: add use cases
section
2020-03-04 17:42:27 +02:00
Aliaksandr Valialkin
032c88561b
app/vminsert/prompush: limit memory usage by pushing promscrape data in smaller blocks
2020-03-03 19:58:54 +02:00
Aliaksandr Valialkin
76036c1897
app/vmagent: add -remoteWrite.maxDiskUsagePerURL
for limiting the maximum disk usage for each -remoteWrite.url
buffer
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin
c31d640eb9
app/vmagent/remotewrite: do not reset empty relabelCtx
2020-03-03 15:01:03 +02:00
Aliaksandr Valialkin
02b55c72dc
app/vmagent: add -remoteWrite.urlRelabelConfig
for applying individual relabeling for each -remoteWrite.url
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308
2020-03-03 13:12:16 +02:00
Aliaksandr Valialkin
7d178a40bd
app/vmselect/prometheus: do not add __name__!=
filter when searching for all the matching metric names via /api/v1/label/__name__/values
with non-empty label filter
...
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin
f9a584b5c1
app/vmagent/remotewrite: yet another typo fix
2020-02-28 20:05:55 +02:00
Aliaksandr Valialkin
b9b46cb8dc
app/vmagent/remotewrite: typo fix
2020-02-28 19:03:16 +02:00
Aliaksandr Valialkin
db6f4e4af1
app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage
2020-02-28 18:58:01 +02:00
Aliaksandr Valialkin
5701b2f7bb
app/vmselect/prometheus: properly pass filter for labelName=__name__
in labelValuesWithMatches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin
18af31a4c2
all: properly split vm_deduplicated_samples_total
among cluster components
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
c458b521a2
app/vmagent: allow setting -httpListenAddr
to empty string in order to disable listening for http requests
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340
2020-02-26 20:58:11 +02:00
Aliaksandr Valialkin
117c76311c
app/vmagent/README.md: list service discovery mechanisms, which will be added soon
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
2020-02-26 19:27:08 +02:00
Aliaksandr Valialkin
be37d762cd
app/vmagent: add -remoteWrite.maxBlockSize
command-line flag for limiting the maximum size of unpacked block to send to remote storage
2020-02-25 19:57:47 +02:00
Aliaksandr Valialkin
4e24839a2c
app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize
2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin
6386aeb1e0
app/vmagent: add ability to accept Influx line protocol data via TCP and UDP
...
Just set `-influxListenAddr` command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin
e453880084
app/vmagent/README.md: mention that vmagent
exposes target statuses at /targets
page
2020-02-25 18:15:58 +02:00
Aliaksandr Valialkin
4c4448b66e
app/vminsert: add /targets
handler, which exposes Prometheus targets defined in -promscrape.config
file
2020-02-25 18:13:11 +02:00
Aliaksandr Valialkin
e1ef72af01
app/vmagent: logo fix
2020-02-25 00:09:19 +02:00
Aliaksandr Valialkin
56c70fe856
app/vmagent: update docs
2020-02-25 00:09:18 +02:00
Aliaksandr Valialkin
e7e4aa5243
app/vmagent/README.md: small fixes
2020-02-24 21:25:38 +02:00
Aliaksandr Valialkin
ae51300973
app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin
e65ec88779
app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-23 18:03:57 +02:00
Yaroslav
a6d0645539
fix rollupOpen(), rollupHigh(), rollupLow() functions ( #328 )
2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00
kreedom
49390b8dbc
[vmalert] integration with AlertManager ( #325 )
2020-02-21 23:15:05 +02:00
Aliaksandr Valialkin
2f55cabaa4
app/vmselect/promql: log when rollupResult cache is cleared
2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin
ec9bf39b5b
app/vmselect: add -search.cacheTimestampOffset
command-line flag
...
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.
This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin
539139391c
app/vmselect: add /internl/resetRollupResultCache
handler for resetting response cache
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 13:58:05 +02:00
kreedom
3c06179184
basic vmalert backbone ( #317 )
...
* basic vmalert backbone
* Resolve code review comments for vmalert backbone
* Second review fixes for vmalert backbone
2020-02-16 20:59:02 +02:00
Aliaksandr Valialkin
7b1c7051a3
app/vmselect: add sort_by_label(q, label)
and sort_by_label_desc(q, label)
functions
...
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin
8466ab0034
all: allow setting flags via environment vars
...
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin
e210cd9da1
lib/storage: move -dedup.minScrapeInterval
flag outside lib/storage, so it doesnt show up in vminsert
in cluster version
2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin
a6c6a2debc
app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8
2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin
78b62dee87
app/vmselect/promql: properly adjust time range for data to select
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin
366693b9f1
app/vmselect: unconditionally offset -step
to rollup_candlestick
. This makes results more consistent
2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin
525101339e
app/vmselect/promql: automatically apply offset -step
to rollup_candlestick
function in order to obtain the expected OHLC results
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin
ada6a3da8d
app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin
36a1ac8360
app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests
is exceeded
...
This will prevent from excess CPU usage for timed out queries.
2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin
834051e5b2
app/vmselect: add a placeholder for /api/v1/metadata
, which could be requested by Grafana
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata
VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.
2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin
42864bb52f
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:46:52 +02:00
Artem Navoiev
a47f292295
[vmalert] add vmalert.png.2
2020-02-02 12:17:19 +02:00
Artem Navoiev
354232b62b
[vmalert] add vmalert.png
2020-02-02 12:16:05 +02:00
Artem Navoiev
28778be0cc
[vmalert] initial
2020-02-02 12:14:09 +02:00
Aliaksandr Valialkin
90cf356ea1
app/vmselect/promql: adjust and
and unless
binary operator handling to be consistent with Prometheus
2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin
c3d86eef96
all: add -dedup.minScrapeInterval
command-line flag for data de-duplication
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
d70ba7eb37
lib/fs: optimize small reads for ReaderAt.MustReadAt
by reading from memory-mapped space instead of reading from file descriptor
...
This should improve performance when reading many small blocks.
2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin
d68546aa4a
lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init
...
This should speed up Search.NextMetricBlock loop for big number of found time series.
2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin
a462355b2f
app/vmselect/promql: add keep_next_value(q)
for filling gaps with the next non-empty value
2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin
371e86194d
app/vminsert: moved -maxInsertRequestSize
command-line flag out of lib/prompb
in order to prevent its inclusion in vmselect
and vmstorage
apps
2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin
adbbc4fa1a
app/vmselect/promql: return expected results from increase()
over the beginning of time series, which start from big value
...
Examples for such counters: OS-level counters for network or cpu stats.
2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin
75ad47a43c
app/victoria-metrics: check for error arg passed to filepath.Walk callback
2020-01-27 20:56:45 +02:00
Aliaksandr Valialkin
6320a19a8c
app/victoria-metrics: remove integration
build tag from tests
...
This simplifies testing with `go test ./app/victoria-metrics` without
the need to remember to pass `-tags=integration` to Go commands.
2020-01-27 20:25:28 +02:00
Aliaksandr Valialkin
2392a359e1
app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin
9df5b2d1c3
app/victoria-metrics: add -selfScrapeInterval
flag for self-scraping /metrics
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180
2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin
6456c93dbb
app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent
2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin
680080887d
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
9773022e50
app/vmselect: mention the original query and time range in error messages
...
This should simplify debugging invalid or heavy queries.
2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin
efc7ad88ec
app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors
2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin
ec9651e266
app/vmselect/prometheus: increase default value -maxExportDuration
to 30 days, since 10 minutes beat users exporting bit amounts of data
2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin
582dd01f42
app/vmselect/promql: add range_over_time(m[d])
function for calculating value range for m
over d
2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin
36973ee975
app/vmselect/promql: add label_match(q, label, regexp)
and label_mismatch(q, label, regexp)
functions for filtering out time series with labels matching the given regexp
2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin
227cf53ef9
app/vminsert: increase default value for -insert.maxQueueDuration
from 30s to 60s
...
This should help catching up with high ingestion rate after VictoriaMetrics restart.
2020-01-18 14:39:36 +02:00
Aliaksandr Valialkin
c9472e4f3a
all: use github.com/klauspost/compress/gzip
instead of compress/gzip
...
`github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`.
This gives better gzip compression and decompression speeds.
2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin
cdcacaea6d
app/vmselect/netstorage: make fmt
2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin
7327adbc86
app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file
...
This should reduce memory usage on systems with more than 8GB RAM.
2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin
d0d258b314
app/vmselect: limit the default value for -search.maxConcurrentRequests
, so it plays well on systems with more than 16 vCPUs
...
A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value.
This will give more chances for executing insert path.
2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin
d88725f133
app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects
2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin
0d7505b00e
all: mention command-line flags used for limiting the incoming request size in error messages
...
This should improve error logs usability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin
2839f4688a
app/vmselect/promql: fix panic on sum(aggr_over_time(...))
with incorrect number of args
2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin
8b14572f70
app/vmselect/promql: add hoeffding_bound_upper(phi, m[d])
and hoeffding_bound_lower(phi, m[d])
functions
...
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin
8eaced8cae
app/vmselect/promql: return continuous values for min_over_time
and max_over_time
when step
is smaller than scrape_interval
2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin
ea231f8167
app/victoria-metrics: adjust integration tests after the commit 99facd71cd6ac151d512cea1df73be91c10c7f83
2020-01-11 00:58:16 +02:00
Aliaksandr Valialkin
46bfdbe6cf
app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time
, max_over_time
, rollup_first
and rollup_last
functions
...
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin
4f0a645f77
vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5
...
This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.
2020-01-10 23:15:15 +02:00
Aliaksandr Valialkin
b829fe5e39
app/vmselect/promql: properly handle aggr(aggr_over_time(...))
2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin
164278151f
app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])
function
...
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin
c4632faa9d
app/vmselect/promql: add tmin_over_time(m[d])
and tmax_over_time(m[d])
functions
...
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin
f358fb72d1
app/{vmbackup,vmrestore}: add backup complete
file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin
d5aeda0e1a
app/vmselect/promql: skip rate
calculation for the first point on time series
2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin
d410c78c7e
app/vmselect/promql: fix calculations for histogram_share
2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin
d64699bb9f
app/vmselect/promql: add missing MetricName into netstorage.Result in tests
2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin
f409f2d050
app/vmselect/promql: add histogram_share(le, buckets)
function
2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin
b1ded7cf9a
app/vmselect/promql: add absent_over_time(m[d])
func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0
app/vmselect/promql: add histogram_over_time(m[d])
rollup function
2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
3e09d38f29
app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time
...
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
a774120460
app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate
for the first point on the time series
...
This should provide better estimation for `rate` in the beginning of time series.
2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin
b5645ccbdf
app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something)
from 100 to 1K, since 100 may result in OOM for high number of time series
2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin
cb3a342882
app/vmselect/promql: add share_le_over_time
and share_gt_over_time
functions for SLI and SLO calculations
2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
6f67e0b56b
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43
Split Extended PromQL parsing to a separate library
2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
d880325cf6
app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range
...
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:
BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin
9e19949c6b
app/vmselect/promql: adjust calculations for rate
and increase
for the first value
...
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin
0455c03cb9
app/vmselect/promql: properly calculate rate
on the first data point
...
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin
5cb8d97743
all: use gozstd instead of pure Go zstd for GOARCH=amd64
2019-12-24 12:42:42 +02:00
Aliaksandr Valialkin
5b75984aa9
app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs
2019-12-23 23:16:11 +02:00