Aliaksandr Valialkin
23bdc1f107
app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series
calls
2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin
c584aece38
app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin
2985077c35
all: consistently use "%w" formatting in fmt.Errorf for wrapped errors
2020-09-23 22:46:34 +03:00
Aliaksandr Valialkin
27500d7d4c
app/vmselect/prometheus: code cleanup after 3ba507000c
2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin
3ba507000c
app/vmselect/prometheus: return timestamps from /api/v1/query
, which match the time
query arg
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin
bed25e3c24
app/vmselect/netstorage: properly pre-allocate space for sbs
2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin
09b0f7c202
app/vmselect/netstorage: release search resources on timeout errors
...
Previously these resources weren't released, which could lead to resource leaks.
2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin
3b1e3a03e0
app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin
94f7d00537
docs/vmagent.md: typo fix
2020-09-21 21:49:22 +03:00
Aliaksandr Valialkin
00b5145c69
app/vmselect/searchutils: fixed tests after 2eb72e09ab
2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin
2eb72e09ab
app/vmselect: use time
value rounded to seconds if it isnt passed to /api/v1/query
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-21 21:24:40 +03:00
Nikolay Khramchikhin
312fead9a2
Add improvements to ec2_sd_discovery ( #775 )
...
* Add improvements to ec2 discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771
role_arn support with aws sts
instance iam_role support
refreshing temporary tokens
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
* changed implementation, removed tests, clean up code
* moved endpoint builder into getEC2APIResponse
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin
1e1a27d803
app/vmalert: remove unneeded UTC() call
...
UTC() doesn't change the underlying timestamp, so the call isn't needed here
2020-09-21 15:55:59 +03:00
Roman Khavronenko
5dffc7a553
vmalert: add support for datasource.lookback
flag ( #779 )
...
New datasource flag `datasource.lookback` defines how far to look into
past when evaluating queries.
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668
2020-09-21 15:53:49 +03:00
Roman Khavronenko
82c3bbce34
vmalert: fix the typo in error message ( #782 )
...
The error will be always nil so no sense in printing it.
2020-09-21 11:34:23 +03:00
Aliaksandr Valialkin
d50165ad59
app/vmagent: increase default value for -remoteWrite.queues
from 1 to 4, since it has been appeared that many users hit this limit
2020-09-18 14:21:54 +03:00
Aliaksandr Valialkin
98d1cd0971
app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar
according to Graphite Metrics API
2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin
7a134b0fd7
app/vmstorage: added -forceMergeAuthKey
command-line flag for protecting /internal/force_merge
endpoint
2020-09-17 14:21:53 +03:00
Aliaksandr Valialkin
1f33dd717f
lib/storage: add /internal/force_merge
handler for running forced compactions on historical per-month partitions
...
This may be useful for freeing up storage space after time series deletion.
See https://victoriametrics.github.io/#force-merge for more details.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin
ab53cb6f7b
app/vmagent: substitute -remoteWrite.url
with secret-url
value in logs, since it may contain sensitive info such as passwords or auth tokens
...
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin
9f79bcf64a
app/vmselect: improve description for -search.maxQueryDuration
2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin
1fec47a289
app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series
2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin
8c3d7c1a59
app/vmselect: typo fix in -search.maxStalenessInterval description
2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin
0e533d1a9c
app/vmselect/promql: support composite durations like Prometheus 2.21 does
...
The following durations are supported now: `1h5m35s` or `1s543ms`
See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
2020-09-11 23:39:13 +03:00
Roman Khavronenko
6ad6480400
vmalert: add Group name as label to generated alerts and timeseries ( #761 )
...
Solves #611
2020-09-11 20:52:56 +01:00
Roman Khavronenko
4cdffb04a4
vmalert: update groups on config reload only if changes detected ( #759 )
...
On config reload event `vmalert` reloads configuration for every group. While
it works for simple configurations, the more complex and heavy installations may
suffer from frequent config reloads.
The change introduces the `checksum` field for every group and is set to md5 hash
of yaml configuration. The checksum will change if on any change to group
definition like rules order or annotation change. Comparing the `checksum` field
on config reload event helps to detect if group should be updated.
The groups update is now done concurrently, so reload duration will be limited by
the slowest group now.
Partially solves #691 by improving config reload speed.
2020-09-11 20:14:30 +01:00
Aliaksandr Valialkin
ca856284e4
app/vmagent: allow setting multiple identical -remoteWrite.url
values
...
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
2020-09-11 15:17:22 +03:00
Aliaksandr Valialkin
a2f647d142
app/vmselect/prometheus: typo fix in the description for -search.latencyOffset
command-line flag
2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin
2380e9b017
app/{vminsert,vmagent}: allow passing timestamp via timestamp
query arg when ingesting data to /api/v1/import/prometheus
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
2020-09-11 13:27:14 +03:00
Aliaksandr Valialkin
f0005c3007
app/vmselect: move Deadline from netstorage to searchutils
...
This removes dependency on netstorage from searchutils.
2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin
2114179e19
app/vmselect: substitute inf values at smooth_exponential with the previous values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin
204ec415b4
app/vmselect: skip infinite values when calculating smooth_exponential
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin
8a8b5a73d3
app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric
2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin
f6bc608e86
app/vmselect: initial implementation of Graphite Metrics API
...
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
9d8fdff6c5
lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps
...
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.
Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin
d7c04db1fc
docs: sync docs for vmalert, vmauth, vmbackup and vmrestore
2020-09-09 21:10:34 +03:00
Aliaksandr Valialkin
11eaa37111
docs/vmagent.md: clarified the case when -remoteWrite.queues
must be tuned
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745
2020-09-08 20:15:27 +03:00
Aliaksandr Valialkin
62919eaf7e
app/vmselect/promql: go fmt
2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin
e6da63dffe
app/vmselect/promql: adjust integrate()
calculations to be more similar to calculations from InfluxDB: attempt #2
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin
8e85b56737
app/vmselect/promql: adjust integrate()
calculations to be more similar to calculations from InfluxDB
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin
c0343a661b
app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3
instead of multiplying by 1e-3
2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin
f4e7e5fb90
app/vmselect/promql: add count_le_over_time(m[d], le)
and count_gt_over_time(m[d], gt)
functions
...
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin
e706e59d49
app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does
2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin
ddabc13796
app/vmagent: properly flush big blocks of data
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741
Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
2020-09-03 12:12:39 +03:00
Aliaksandr Valialkin
7a839b461f
app/vmagent: fix data race when accessing writeRequest.lastFlushTime
2020-09-03 12:12:37 +03:00
Nikolay Khramchikhin
764b3d4fda
changed vmalert behaviour ( #738 )
...
* VMAlert start with empty rules dir
There are some applications (operator for instance), that generates alerts configuration at runtime
and vmalert must start correctly without rules to support this behaviour.
Later application will add rules files and send SIGHUP to vmalert,
which will trigger reading rules files and start rules exectuion.
Removing rules files with SIGHUP signal must stop rules execution and
vmalert will wait for new rules.
* imports sorted
* added test cases for empty rules, removed blank line
* fixed imports conflict
* updated tests
2020-09-03 11:04:42 +03:00
Aliaksandr Valialkin
5f16ceb294
app/vmalert: imrovements over 3f932c2db1
2020-09-03 01:00:55 +03:00
DexterZhang
3f932c2db1
feat: spread load of rule evaluation by group when starting new groups ( #724 )
...
* feat: spread load of rule evaluation by group when starting new groups
* review: reduce the resulting diff.
* Update app/vmalert/group.go
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-03 00:58:54 +03:00
Aliaksandr Valialkin
f41b36bb9a
app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats
...
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
2020-09-02 19:43:21 +03:00
Aliaksandr Valialkin
5af777469a
app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url
2020-08-30 21:40:22 +03:00
Aliaksandr Valialkin
2149733bd2
app/vmagent: apply sane limits to -remoteWrite.queues
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
2020-08-30 21:25:37 +03:00
Roman Khavronenko
10601bc652
vmalert: update -rule
flag description to enforce quotes using ( #709 )
...
Description for `-rule` flag uses as example specific chars like asterisks
which could be interpreted wrong by different shells. To avoid this, description
now contains quoted flag values.
See also #708
2020-08-20 22:36:38 +01:00
Roman Khavronenko
f5f59896ec
lib/decimal: rename significant decimal digits
to significant figures
( #698 )
...
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures ) rounding
applied to all significant figures, not just decimal ones.
2020-08-16 17:21:35 +03:00
Aliaksandr Valialkin
147c35ebd4
all: allow using KB
, MB
, GB
, KiB
, MiB
and GiB
suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin
7d89fafe1a
app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)
2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin
cd96248480
docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion
2020-08-14 20:47:57 +03:00
Aliaksandr Valialkin
184670fb9b
docs: update docs
2020-08-14 19:13:42 +03:00
Aliaksandr Valialkin
576da0fe46
app/{vminsert,vmagent}: improve documentation for -influxListenAddr
command-line flag
2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin
c402265e88
all: support %{ENV_VAR}
placeholders in yaml configs in all the vm* components
...
Such placeholders are substituted by the corresponding environment variable values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
2020-08-13 17:15:25 +03:00
Aliaksandr Valialkin
ef7e2af8f5
app: respect CPU limits set via cgroups
...
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-11 22:59:19 +03:00
Aliaksandr Valialkin
e969ef2639
app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export
if max_rows_per_line
is set to non-zero value
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin
8adba82c02
app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores
...
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin
f3d33e23c9
app/vmstorage: improve error logging when the request times out
2020-08-10 13:23:26 +03:00
Roman Khavronenko
0be5b09fb4
app/vmalert: extend metrics set exported by vmalert
#573 ( #654 )
...
* app/vmalert: extend metrics set exported by `vmalert` #573
New metrics were added to improve observability:
+ vmalert_alerts_pending{alertname, group} - number of pending alerts per group
per alert;
+ vmalert_alerts_acitve{alertname, group} - number of active alerts per group
per alert;
+ vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error
during prev execution, is 0 if no errors happened;
+ vmalert_recording_rules_error{recording, group} - is 1 if recording rule
ended up with error during prev execution, is 0 if no errors happened;
* vmalert_iteration_total{group, file} - now contains group and file name labels.
This should improve control over specific groups;
* vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups;
Some collisions for alerts and recording rules are possible, because neither
group name nor alert/recording rule name are unique for compatibility reasons.
Commit contains list of TODOs for Unregistering metrics since groups and rules
are ephemeral and could be removed without application restart. In order to
unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13
* app/vmalert: extend metrics set exported by `vmalert` #573
The changes are following:
* add an ID label to rules metrics, since `name` collisions within one group is
a common case - see the k8s example alerts;
* supports metrics unregistering on rule updates. Consider the case when one rule
was added or removed from the group, or the whole group was added or removed.
The change depends on https://github.com/VictoriaMetrics/metrics/pull/16
where race condition for Unregister method was fixed.
2020-08-09 09:41:29 +03:00
ofen
81746d14b9
401 Unauthorize HTTP error added ( #681 )
...
401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication ]
2020-08-09 09:38:41 +03:00
Aliaksandr Valialkin
716754fae6
app/vmselect/promql: properly handle -n^m
like Prometheus does
...
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin
bb61a4769b
app/vmselect/promql: remove metric name after applying clamp_min
and clamp_max
functions in order to be consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin
ac45082216
app/vmselect/promql: remove metric name after applying ceil
, floor
and round
functions in order to be more consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin
e5202a4eae
app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus
...
Rollup functions:
- avg_over_time
- min_over_time
- max_over_time
- quantile_over_time
This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin
68e4f40a72
app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2
like Prometheus does
...
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin
bc8381613d
app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names
2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
3f85c06b65
app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block
...
This should improve query performance on systems with big number of CPU cores (16 and more)
2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin
d20c2156e4
app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans
2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin
dbbdfbe7ee
app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"}
to vm_cache_entries{type="storage/prefetchedMetricIDs"}
to be consistent with other vm_cache_entries
metrics
2020-08-06 16:34:24 +03:00
Aliaksandr Valialkin
8f16388428
lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2
...
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin
c25b0c2cd5
app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
2020-08-04 21:00:29 +03:00
Aliaksandr Valialkin
5d0c37bec0
app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration
2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin
e6eee2bebf
app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d])
and zscore(q) by (...)
2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin
509d12643b
app/vmselect: show X-Forwarded-For
contents on /api/v1/status/active_queries
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin
d01f3c1943
all: add mssing APP_NAME to vm*-GOARCH builds
2020-07-31 13:42:18 +03:00
Aliaksandr Valialkin
3f498cf2dc
docs/{vmagent,vmalert}: add instruction on how to build for ARM
2020-07-31 09:25:22 +03:00
Aliaksandr Valialkin
f0c678c41b
app/vmselect: do not adjust start
and end
query args passed to /api/v1/query_range
when -search.disableCache
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin
e7959094f6
lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected
...
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.
Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin
328b52e5ff
app/vmselect/promql: return non-empty value from rate_over_sum(m[d])
even if a single data point is located in the given [d]
window
...
Just divide the data point value by the window duration in this case.
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
700737c181
app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase()
is handled in another way now
...
See e00cfc854d
for the approach used now.
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
2f735f112d
app/vmselect/promql: fill gaps with 0 in rate_over_sum
response when the last value before the selected time window isnt empty
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
0f63da3698
app/{vmagent,vminsert}: properly preserve db
tag from query string passed to Influx line protocol query
...
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
2020-07-28 21:25:19 +03:00
Aliaksandr Valialkin
62ed38c6f0
app/vmagent/remotewrite: add missing resp.Body.Close()
after pushing data to remote storage
...
Missing body close could disable HTTP keep-alive connections.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
2020-07-28 21:00:15 +03:00
Aliaksandr Valialkin
79c30cf4cb
app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries
page for every query
2020-07-28 15:13:08 +03:00
Roman Khavronenko
2f1e7298ce
app/vmalert: support external.label
to specify global labelset for all rules #622 ( #652 )
...
`external.label` flag supposed to help to distinguish alert or recording rules
source in situations when more than one `vmalert` runs for the same datasource
or AlertManager.
2020-07-28 14:20:31 +03:00
Aliaksandr Valialkin
0da202023b
app/vmselect/promql: return empty values from group()
if all the time series have no values at the given timestamp
...
This aligns `group()` behaviour to Prometheus
2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin
cb8c6908dc
app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage
...
Previously the request body was already consumed before the retry, so this led to the following error:
http: ContentLength=... with Body length 0
2020-07-27 17:32:46 +03:00
Aliaksandr Valialkin
894dcb7c1c
app/vmselect/promql: improve further the accuracy of buckets_limit() function
...
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin
215eba0b82
app/vminsert: flush bufs if needed after the current row is added
...
Previously the data for the added row could be overwritten by the flush
before the row addition is complete.
2020-07-26 12:10:11 +03:00
Aliaksandr Valialkin
edb1eca6f1
app/vmselect/promql: avoid dropping inf
bucket in buckets_limit
...
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin
97b6f5d223
app/vmselect/promql: optimize buckets_limit(k, buckets)
for big number of buckets
2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin
a090627059
app/vminsert: limit memory usage when ingesting data in big packets
2020-07-24 23:32:40 +03:00
Aliaksandr Valialkin
bb161497cf
app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets)
function
...
Now it properly merges the bucket with the previous bucket after deletion.
2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin
994fa2f3bf
app/vmselect/promql: add buckets_limit(k, buckets)
function, which limits the number of buckets per time series to k
...
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin
e151c5c644
app/vmselect: fix tests for rate_over_sum
2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin
3107c633e3
app/vmselect/promql: typo fix after 3e557c9861
2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin
3e557c9861
app/vmselect/promql: add rate_over_sum(m[d])
function to MetricsQL, which returns rate over sum of m
values over d
duration
...
Something like `sum_over_time(m[d]) / d`, but more accurate.
2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin
b1f6843bd0
app/vmselect/promql: allow setting [d]
window smaller than the interval between raw points for avg_over_time
...
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin
039c9d2441
lib/storage: respect -search.maxQueryDuration
when searching for time series in inverted index
...
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823
lib/storage: add more fine-grained pace limiting for search
2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
461481fbdf
app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery
2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin
4c8b49b193
app/vminsert: export vm_relabel_metrics_dropped_total
metric that shows the number of metrics dropped due to relabeling
2020-07-23 14:57:53 +03:00
Aliaksandr Valialkin
e79de9774b
app/vmselect: typo fix after 34563916f7
2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin
34563916f7
app/vmselect: reduce memory usage when querying big number of time series with long labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin
9257eee982
app/vminsert: do not call ApplyRelabeling function if relabeling is disabled
...
This should reduce CPU usage a bit when `-relabelConfig` isn't set
2020-07-23 13:39:44 +03:00
Aliaksandr Valialkin
6f05c4d351
lib/storage: improve prioritizing of data ingestion over querying
...
Prioritize also small merges over big merges.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin
2f612e0c67
app/vminsert: fix relabeling for metrics ingested via Influx line protocol
...
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin
4f526cc816
app/vmselect/prometheus: support d
, w
and y
suffixes for durations passed to step
in /api/v1/query_range
like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin
dfb113f175
app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries
...
This should slightly reduce load on GC when processing queries that touch big number of time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.
2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin
31ae5911a8
app/vmagent: add -remoteWrite.decimalPlaces
command-line flag, which may be used for reducing disk space usage on the remote storage
2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin
caa2952aa6
app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query
2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin
e00cfc854d
app/vmselect/promql: skip the first value in time series passed to increase()
if it exceeds by more than 10x the delta between the next value and the first value
...
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin
b9c8f6bf34
app/vmselect: log the total available memory for concurrent requests on not enough memory
errors
...
This should simplify root cause analysis
2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin
ad6290953c
app/vmagent: add -remoteWrite.proxyURL
command-line option
...
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
2020-07-20 19:28:49 +03:00
Aliaksandr Valialkin
efcbb51968
docs/vmagent.md: sync with app/vmagent/README.md
2020-07-20 17:08:34 +03:00
Roman Khavronenko
ed0df37ee7
app/vmagent: mention grafana dashboard in README ( #639 )
2020-07-20 17:07:27 +03:00
Aliaksandr Valialkin
11be704109
app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig
entries
2020-07-20 15:49:27 +03:00
Aliaksandr Valialkin
5a4675c528
app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range
...
This matches Prometheus behaviour.
This should fix https://github.com/jacksontj/promxy/issues/329
2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin
ecb1b2564a
app/vmselect/promql: add mode()
aggregate function
2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin
b35cb293f5
lib/httpserver: log remote address in error message from httpserver.Errorf
...
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
1c641037e8
app/vmselect/promql: add mode_over_time(m[d])
function
...
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin
6b5ad535ae
app/vmselect/promql: optimize group(rollup(m))
calculations
2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin
8949d65ad1
app/vmselect/promql: check that any()
doesn't touch metric name
2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin
aa5d88055d
app/vmselect/promql: add group()
aggregate function to MetricsQL
...
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin
df01836818
app/vmselect/promql: keep all labels for time series from any()
call
2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin
8c14ca93fa
app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call
...
Previously this could lead to `out of range` panic
2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin
e4e1cd1de2
app/vmselect: fix nil pointer dereference
panic when unsuccessfully querying vmstorage
2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin
ed7580ad22
app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf
when wrapping errors
2020-07-15 13:56:47 +03:00
Aliaksandr Valialkin
328814ee60
docs/vmagent.md: make filtering rules for init container pods less confusing
2020-07-14 20:32:47 +03:00
Aliaksandr Valialkin
b442a42d8e
app/vmagent/remotewrite: return proper value from tssRelabelPool.New
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-14 14:29:20 +03:00
Aliaksandr Valialkin
4081e2295e
app/{vminsert,vmagent}: add -influxSkipMeasurement
command-line flag for using field name as metric name
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin
25f80d320b
app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time
...
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin
55d83e777d
app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a
over 619b0a25c9
2020-07-13 21:40:38 +03:00
faceair
1033dc7e2a
fix empty response template ( #617 )
2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin
619b0a25c9
docs/vmagent.md: sync with app/vmagent/README.md
2020-07-13 21:25:11 +03:00
ofen
666c795b98
Update README.md ( #621 )
...
Troubleshooting section updated to help out with duplicate targets detection
2020-07-13 21:18:54 +03:00
Aliaksandr Valialkin
a730b3f6a1
app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig
options are set
...
Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race
and improper relabeling. Now each goroutine has its own copy of tss during relabeling.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-10 15:16:59 +03:00
Aliaksandr Valialkin
508ad46e0e
app/vmagent/remotewrite: typo fix in -remoteWrite.showURL
help message
2020-07-10 14:07:08 +03:00
Aliaksandr Valialkin
cba820e390
app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus
2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin
9c350bc20d
app/vmselect/promql: add missing tests for ifnot
binary operation
2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin
256fd9a87e
app/vmselect/promql: refactor implementations for and
and unless
binary operations, so they are closer to or
implementation
2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin
2d9b3ad5b3
app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function
2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin
742da690f4
app/vmselect: add /api/v1/status/active_queries
page with the list of currently running queries
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575
2020-07-08 18:55:38 +03:00
DexterZhang
99f54e44ff
feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch ( #598 )
2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin
6daa5f7500
lib/storage: prioritize data ingestion over heavy queries
...
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.
Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2020-07-05 19:42:05 +03:00