Commit Graph

1518 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
e0422365c0
docs/CHANGELOG.md: clarify the change at e3ef3df938
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
2023-08-11 05:43:58 -07:00
hagen1778
f847e2f91d
docs: mention honor_timestamps change in changelog
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 05:41:10 -07:00
Zakhar Bessarab
bffec2fc02
{vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency (#4809)
{vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency

Sync behavior for dropInput and keepInput flags between single-node and vmagent.
Fix vmagent not respecting dropInput flag and reverse logic for keepInput.
2023-08-11 05:40:06 -07:00
Yury Molodov
cf0077b552
vmui: allow displaying the full error message on click (#4760)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4719
2023-08-11 04:59:34 -07:00
Haleygo
404c330c5a
docs: add changelog for 4c815ed59b (#4805) 2023-08-11 04:58:27 -07:00
Roman Khavronenko
4c91773a15
vmalert: cleanup config reload metrics handling (#4790)
* rename `configErr` to `lastConfigErr` to reduce confusion
* add tests to verify metrics and msg are set properly
* fix mistake when config success metric wasn't restored after an error

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:56:21 -07:00
hagen1778
05b4fbf0b5
dashboards: correctly calculate Bytes per point value
Correctly calculate `Bytes per point` value for single-server and cluster VM dashboards.
Before, the calculation mistakenly accounted for the number of entries in indexdb in
denominator, which could have shown lower values than expected.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:53:56 -07:00
Roman Khavronenko
7220f70506
security: bump go version from 1.20.6 to 1.20.7 (#4773)
The update includes a security fix to the crypto/tls package,
as well as bug fixes to the assembler and the compiler.

See the list of issues addressed in Go1.20.7 here:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:44:01 -07:00
hagen1778
a7f0b8436c
dashboards: add panels for absoulte value of mem and cpu usage by vmalert
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4627

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:43:01 -07:00
hagen1778
f2b06484f2
alerts: move ConcurrentFlushesHitTheLimit alert to health alerts
The `ConcurrentFlushesHitTheLimit` could be related to components like
vminsert, vmstorage, vm-single-node and vmagent. Moving this alert
to the `health` section of alerts will be benefitial for all components
and will remove the duplicates from single/cluster alerts.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:39:28 -07:00
hagen1778
be7001351c
alerts: add docs section for the full list of alerting rules
The change also includes update of all references in other docs
to the alerting rules.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:38:30 -07:00
hagen1778
2f05be37b3
dashboards: add Concurrent inserts panel to vmagent's dasbhoard
The new panel supposed to show whether the number of concurrent
inserts processed by vmagent isn't reaching the limit.
The panel contains recommendation what to do if limit is reached.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:36:40 -07:00
hagen1778
d06c224d2e
docs: follow-up after df37a47d4b
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4415
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:32:36 -07:00
Yury Molodov
d443ab5ae1
vmui: display partial response warning (#4742)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4721
2023-08-11 04:17:27 -07:00
Dmytro Kozlov
dabce3cc5d
app/vmctl: add flag where use can define path to the source remote read protocol (#4744)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655
2023-08-11 03:56:34 -07:00
Roman Khavronenko
5a5785292c
vmctl: allow disabling binary export protocol (#4716)
Binary export API protocol can be disabled via `-vm-native-disable-binary-protocol` cmd-line flag when migrating data from VictoriaMetrics. Disabling binary protocol
can be useful for deduplication of the exported data before ingestion.
For this, deduplication need to be configured at `-vm-native-src-addr` side
and `-vm-native-disable-binary-protocol` should be set on vmctl side.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 03:19:20 -07:00
Zakhar Bessarab
3099114c7c
docs: add changelog entry for #4704 (#4753)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-08-11 03:17:02 -07:00
Dmytro Kozlov
8d0576c714
app/vmctl: add support the week step for time-based chunks (#4743)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738
2023-07-31 08:00:11 -07:00
Roman Khavronenko
02dba5f728
vmalert: remove deprecated in v1.79.0 web links with */status suffix (#4747)
Links of form `/api/v1/<groupID>/<alertID>/status` were deprecated
in favour of `/api/v1/alerts?group_id=<>&alert_id=<>` links in
v1.79.0. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825

This change removes code responsible for deprecated functionality.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 07:58:37 -07:00
Roman Khavronenko
85eb62a2ec
vmalert: remove deprecated in v1.61.0 -rule.configCheckInterval (#4745)
Use `-configCheckInterval` command-line flag instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 07:55:54 -07:00
hagen1778
e716ae1921
docs: mention 3f6efab6ae in changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 07:53:32 -07:00
hagen1778
7a9c211b36
docs: remove anchors from the 1.92 release
Adding anchors to the 1.92 changelog breaks consistency
of navigation section at https://docs.victoriametrics.com/CHANGELOG.html
All other releases do not have subsections, so should 1.92.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 07:47:22 -07:00
Aliaksandr Valialkin
693866a516
docs/CHANGELOG.md: move bugfix description to tip chapter, since it isnt released yet
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
2023-07-28 23:01:51 -07:00
Aliaksandr Valialkin
3e62c71e8c
lib/promscrape: add a comment why honor_timestamps is set to false by default
This should prevent from returning it back to true in the future

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
2023-07-28 21:36:55 -07:00
Aliaksandr Valialkin
ee98f9ae66
lib/promscrape: use local scrape timestamp for scraped metrics unless honor_timestamps: true is set explicitly
This fixes the case with gaps for metrics collected from cadvisor,
which exports invalid timestamps, which break staleness detection at VictoriaMetrics side.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697 ,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799
and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1656540535

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1773
2023-07-28 21:11:46 -07:00
Zakhar Bessarab
c5c539f973
docs/CHANGELOG.md: cut v1.92.1 (#4735)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 8f257889cc)
2023-07-28 11:09:56 +02:00
Roman Khavronenko
303d3616ec
vmalert: revert unittest feature (#4734)
* Revert "vmalert: unittest support stale datapoint (#4696)"

This reverts commit 0b44df7ec8.

* Revert "docs: specify min version and limitations for vmalert's unit tests"

This reverts commit a24541bd

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "vmalert: init unit test (#4596)"

This reverts commit da60a68d

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs: mention unittest revert in changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 9f1b9b86cc)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-28 11:09:52 +02:00
Aliaksandr Valialkin
174f6793f3
docs/CHANGELOG.md: delimit changes from update notes 2023-07-27 17:06:37 -07:00
Aliaksandr Valialkin
fd4b10330a
docs/CHANGELOG.md: cut v1.92.0 2023-07-27 14:55:52 -07:00
Aliaksandr Valialkin
1f30f53df2
lib/promscrape/discovery: close unused HTTP connections to service discovery servers
This should prevent from connection leaks

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724
2023-07-27 14:47:55 -07:00
Nikolay
85de94e85c
lib/protoparser: adds opentelemetry parser (#2570)
* lib/protoparser: adds opentelemetry parser
app/{vmagent,vminsert}: adds opentelemetry ingestion path

Adds ability to ingest data with opentelemetry protocol
protobuf and json encoding is supported
data converted into prometheus protobuf timeseries
each data type has own converter and it may produce multiple timeseries
from single datapoint (for summary and histogram).
only cumulative aggregationFamily is supported for sum(prometheus
counter) and histogram.

Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

updates deps

fixes tests

wip

wip

wip

wip

lib/protoparser/opentelemetry: moves to vtprotobuf generator

go mod vendor

lib/protoparse/opentelemetry: reduce memory allocations

* wip

- Remove support for JSON parsing, since it is too fragile and is rarely used in practice.
  The most clients send OpenTelemetry metrics in protobuf.
  The JSON parser can be added in the future if needed.
- Remove unused code from lib/protoparser/opentelemetry/pb and lib/protoparser/opentelemetry/proto
- Do not re-use protobuf message between ParseStream() calls, since there is high chance
  of high fragmentation of the re-used message because of too complex nested structure of the message.

* wip

* wip

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-27 13:37:15 -07:00
Aliaksandr Valialkin
b3e93afd17
docs/CHANGELOG.md: add a link to Pushgateway protocol in the bugfix description for 74237ce5c0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692
2023-07-27 13:11:25 -07:00
Alexander Marshalov
74237ce5c0
fixed label values decoding for pushgateway compatibility (#4727)
Fixed decoding of label values with slash for pushgateway and prometheus golang client compatibility + added some tests. (#4962)
2023-07-27 13:03:48 -07:00
Haleygo
3c297e0253
vmalert: add keep_firing_for field for alerting rule (#4669)
vmalert: support `keep_firing_for` field for alerting rule

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4529

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-27 13:00:45 -07:00
Aliaksandr Valialkin
fa295c7daa
app/vmagent: add ability to shard outgoing data among multiple remote storage systems
Add -remoteWrite.shardByURL command-line flag, which instructs vmagent to spread evenly
outgoing time series data among the configured remote storage systems specified via -remoteWrite.url .

Samples for the same time series go to the same -remoteWrite.url . This allows building horizontally
scalable stream aggregation when samples for counter and histogram series must be aggregated
by the same second-level vmagent instance.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637
2023-07-24 18:18:04 -07:00
Aliaksandr Valialkin
6e43664e24
lib/promrelabel: add support for a list of series selectors at IfExpression
This makes possible specifying a list of series selectors at the following places:

- Inside `if` option at relabeling rules
- Inside `match` option at stream aggregation rules

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635
2023-07-24 17:09:59 -07:00
Aliaksandr Valialkin
c049778ad1
lib/streamaggr: follow-up for 736197179e
- Use a byte slice instead of a map for tracking indexes for matching series.
  This improves performance, since access by slice index is faster than access by map key.
- Re-use the byte slice for tracking indexes for matching series.
  This removes unnecessary memory allocations and improves stream aggregation performance a bit.
- Add an ability to return to the previous behvaiour by specifying -remoteWrite.streamAggr.dropInput command-line flag.
  In this case all the input samples are dropped when stream aggregation is enabled.
- Backport the new stream aggregation behaviour from vmagent to single-node VictoriaMetrics when -streamAggr.config
  option is set.
- Improve docs regarding this change at docs/CHANGELOG.md
- Document the new behavior at docs/stream-aggregation.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4575
2023-07-24 17:06:09 -07:00
Zakhar Bessarab
470afac5ff
{lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag (#4575)
* {lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag

Changes default behaviour of keepInput flag to write series which did not match any aggregators to the remote write.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmagent/remotewrite/remotewrite.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-24 16:34:38 -07:00
Nikolay
30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652)
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
Zakhar Bessarab
2fe33b3d97
app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687)
* app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries

Fixes construction of URL parameters for graphite render to allow overriding "from" parameter.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmalert/datasource/graphite: update flow for building URL parameters

Makes flow of building URL parameters same as Prometheus datasource has:
1) Setting all default values
2) Merging those values with provided `extraParams`

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-22 14:18:52 -07:00
Aliaksandr Valialkin
324a3c5288
lib/promscrape: follow-up after 6aa50ca954
- Improve docs
- Hide `debug relabeling` column when -promscrape.dropOriginalLabels command-line flag is set
- Inline the code from the added template functions, since the code is harder to follow
  with the template functions, especially when these functions have misleading names.
  Also, these functions are used only in one place, e.g. they do not reduce the amounts of code.
- Hide `click to show original labels` title at `labels` column when original labels aren't available.
- Show the reason on whey original labels aren't available at /service-discovery page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597
2023-07-20 21:54:09 -07:00
Aliaksandr Valialkin
c921bc0833
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4294
2023-07-20 21:53:51 -07:00
Alexander Marshalov
9ba03b4838
allow configuring staleness interval in stream aggregation (#4667) (#4670)
---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-20 21:47:29 -07:00
Haleygo
939c8b8372
vmalert: init unit test (#4596)
vmalert: support unit tests

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:19:45 -07:00
Dmytro Kozlov
f0d8f77e6d
app/vmagent: fix creating target id if --promscrape.dropOriginalLabels flag was used (#4616)
* app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used

* app/vmagent: hide links if OriginalLabels was dropped

* app/vmagent: update CHANGELOG.md and added information to the docs

* app/vmagent: fix comments
2023-07-20 19:21:41 -07:00
Yury Molodov
3ad80e281f
vmui: add Active Queries page (#4653)
* feat: add page to display a list of active queries (#4598)

* app/vmagent: code formatting

* fix: remove console

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-07-19 16:02:58 -07:00
Aliaksandr Valialkin
a3c8f902c1
vendor: update github.com/VictoriaMetrics/metricsql from v0.60.0 to v0.61.1
This adds support for passing durations via WITH template vars:

- `WITH (w = 5m) m[w]` is transformed to `m[5m]`
- `WITH (f(w, step, off) = m[w:step] offset off) f(5m, 10s, 1h)` is transformed to `m[5m:10s] offset 1h`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4025
Updates https://github.com/VictoriaMetrics/metricsql/issues/12

See also the initial implementation by @lujiajing1126 at https://github.com/VictoriaMetrics/metricsql/pull/13
2023-07-19 15:04:38 -07:00
hagen1778
7703059746
docs: typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-19 14:49:31 -07:00
Roman Khavronenko
debe1793bb
vmalert: follow-up after d4ac4b7813 (#4659)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-18 16:03:28 -07:00
hagen1778
456a2e70fd
docs: mention change from 6f3fee197e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-18 16:01:59 -07:00