Aliaksandr Valialkin
17b87725ed
lib/promscrape/discovery/kubernetes: reduce memory usage when Kubernetes service discovery is configured on a big number of scrape jobs
...
Previously vmagent was creating a separate Kubernetes object cache per each scrape job.
This could result in increased memory usage when monitoring a Kubernetes cluster with big number of objects (pods / nodes / services, etc.)
as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
Now it uses a shared map of scrape objects across multiple scrape jobs.
2021-03-05 17:29:55 +02:00
Aliaksandr Valialkin
a14053ffa0
app/vmselect/promql: add histogram_avg()
, histogram_stddev()
and histogram_stdvar()
functions to MetricsQL
2021-03-04 14:12:07 +02:00
Aliaksandr Valialkin
423cd981fb
lib/promscrape: add -promscrape.cluster.replicationFactor
command-line flag for replicating scrape targets among vmagent
instances in the cluster
2021-03-04 10:20:15 +02:00
Aliaksandr Valialkin
3a68b94487
docs/CHANGELOG.md: cut v1.55.1 release
2021-03-03 11:49:10 +02:00
Aliaksandr Valialkin
4c3ef78c05
docs/CHANGELOG.md: mention recent bugfixes from commits 7906316741
and e154f4a644
2021-03-03 10:50:31 +02:00
Aliaksandr Valialkin
1252ca44d8
docs/CHANGELOG.md: cut v1.55.0
2021-03-02 21:37:58 +02:00
Aliaksandr Valialkin
22b1941cfc
lib/promscrape/discovery/ec2: follow-up after f6114345de
2021-03-02 13:46:26 +02:00
Aliaksandr Valialkin
937f382938
lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format
...
Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.
2021-03-02 13:21:43 +02:00
Aliaksandr Valialkin
019d8e88d8
lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as foo; 1 2
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100
2021-03-01 17:16:35 +02:00
Aliaksandr Valialkin
2c43e846a9
docs/CHANGELOG.md: mention the out of range panic bugfix d6a41b6ea2
2021-03-01 16:54:24 +02:00
Aliaksandr Valialkin
732e729ef9
docs/CHANGELOG.md: mention the issue related to using Kubernetes watch API for service discovery
2021-03-01 01:42:11 +02:00
Aliaksandr Valialkin
cb943f35c7
app/vmagent: remove data race when applying rate limits to -remoteWrite.url
with multiple queues
2021-03-01 00:29:07 +02:00
Aliaksandr Valialkin
1da1d502a8
docs/CHANGELOG.md: mention about https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1074
2021-02-28 19:31:44 +02:00
Aliaksandr Valialkin
9fc7726d84
lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls
...
This should fix the `http: superfluous response.WriteHeader call` issue
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078
2021-02-28 19:22:50 +02:00
Aliaksandr Valialkin
e5ca8ac0db
lib/promscrape: add ability to spread scrape targets among multiple vmagent instances
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
2021-02-28 18:41:08 +02:00
Aliaksandr Valialkin
03903c1176
docs/CHANGELOG.md: mention 317b0cbed2
2021-02-28 14:02:49 +02:00
Aliaksandr Valialkin
7cc3d96a41
lib/fs: follow-up after f3a03c4164
2021-02-27 01:01:47 +02:00
Aliaksandr Valialkin
975dac9086
docs/CHANGELOG.md: mentioned a bugfix with extra_label
handling during caching query results
...
Related to 186c078fac
2021-02-27 00:20:29 +02:00
Aliaksandr Valialkin
8683ea85e6
lib/fs: properly handle stale NFS file handle
error during file deletion
...
This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.
2021-02-26 23:25:14 +02:00
Aliaksandr Valialkin
d86e9b49c4
app/vmselect/promql: increase accuracy for buckets_limit()
function for small limits by skipping the first and the last buckets during merge
...
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
2021-02-26 22:56:36 +02:00
Aliaksandr Valialkin
c8f2f9b2e8
lib/promscrape: cleanup after 9b2246c29b
...
Main points:
* Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit
* Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001
* Corner case fixes.
2021-02-26 16:54:05 +02:00
Aliaksandr Valialkin
98854e5f2b
app/vmselect: add sign(q)
and clamp(q, min, max)
functions, which will be added in the upcoming Prometheus release
...
See https://twitter.com/roidelapluie/status/1363428376162295811
The `last_over_time(m[d])` function already exists in MetricsQL.
2021-02-24 17:24:56 +02:00
Aliaksandr Valialkin
f050e3f492
docs/CHANGELOG.md: mention about a bugfix from 4805b80977
2021-02-24 11:49:10 +02:00
Aliaksandr Valialkin
f4135b0d14
app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero
le="+Inf"` bucket like Prometheus does
2021-02-24 00:42:22 +02:00
Aliaksandr Valialkin
fa03e0d210
app/vmselect/promql: add increase_pure()
function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
2021-02-22 19:14:15 +02:00
Aliaksandr Valialkin
dd1e53b119
lib/promrelabel: optimize relabeling performance for common cases
2021-02-22 00:51:13 +02:00
Aliaksandr Valialkin
ff5bbc4b88
lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric
2021-02-21 23:21:42 +02:00
Aliaksandr Valialkin
901e12024d
vendor: update github.com/VictoriaMetrics/metrics from v1.14.0 to v1.15.0
...
The v1.15.0 exports the following additional metrics:
process_io_read_bytes_total - the number of bytes read via io syscalls such as read and pread
process_io_written_bytes_total - the number of bytes written via io syscalls such as write and pwrite
process_io_read_syscalls_total - the number of read syscalls such as read and pread
process_io_write_syscalls_total - the number of write syscalls such as write and pwrite
process_io_storage_read_bytes_total - the number of bytes read from storage layer
process_io_storage_written_bytes_total - the number of bytes written to storage layer
These metrics can be used for monitoring process io
2021-02-21 22:54:00 +02:00
Aliaksandr Valialkin
c2678754e4
app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-19 00:31:34 +02:00
Aliaksandr Valialkin
f26162ec99
lib/promscrape: add scrape_align_interval config option into scrape config
...
This option allows aligning scrapes to a particular intervals.
2021-02-18 23:53:44 +02:00
Aliaksandr Valialkin
9c70c1f21f
app/vmselect/promql: reduce the probability of duplicate time series
errors when querying Kubernetes metrics
2021-02-18 22:07:29 +02:00
Aliaksandr Valialkin
5e341ccb59
docs/CHANGELOG.md: cut v1.54.1
2021-02-18 19:09:59 +02:00
Aliaksandr Valialkin
a537c4f602
lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters
...
Example: `node_cpu_seconds_total{mode!="idle"}`
2021-02-18 18:46:36 +02:00
Aliaksandr Valialkin
a7697cc88b
docs/CHANGELOG.md: cut v1.54.0
2021-02-18 14:52:38 +02:00
Aliaksandr Valialkin
edcdc39eb3
app/vmagent/remotewrite: cleanup after 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-17 21:42:55 +02:00
Aliaksandr Valialkin
4b110fa21c
docs/CHANGELOG.md: mention that prod binaries are built now with Go1.16
...
This is a follow-up for 4edfe76bef
2021-02-17 21:06:59 +02:00
Aliaksandr Valialkin
bea2f86b7b
docs/CHANGELOG.md: document new per-tenant metrics
2021-02-16 23:34:24 +02:00
Aliaksandr Valialkin
9f5ac603a7
lib/storage: reduce the minimum supported retention for inverted index from one month to one day
2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
38d7e96602
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_*
and __meta_kuberntes_endpoints_annotation_*
labels to role: endpoints
...
This syncs kubernetes SD with Prometheus 2.25
See 617c56f55a
2021-02-15 02:51:16 +02:00
Aliaksandr Valialkin
c727d2219b
lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters.
...
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
2021-02-14 23:38:14 +02:00
Aliaksandr Valialkin
73c95c4e5b
docs/CHANGELOG.md: mention about fixed multiarch build for Docker images
...
Related commit: f9902b3372
2021-02-12 15:23:49 +02:00
Aliaksandr Valialkin
1e38ad6d20
app/vmauth: add ability to route requests from a single users to multiple targets depending on the requested path
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1064
2021-02-11 12:41:16 +02:00
Aliaksandr Valialkin
04faea8b45
app/vmselect: parallelize q1 <binary_op> q2
queries by running q1
and q2
in parallel
...
This should reduce query execution times.
2021-02-10 22:59:39 +02:00
Aliaksandr Valialkin
553016ea99
lib/storage: disable composite index usage when querying old data
2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
c7dccebaef
lib/storage: optimize search by label filters matching big number of time series
2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
a4140de9e6
lib/mergeset: unconditionally cache indexdb blocks
...
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
2021-02-09 00:47:50 +02:00
Aliaksandr Valialkin
4dca03501b
docs/CHANGELOG.md: mention about a bugfix for timezone data from df0cda3ab9
2021-02-08 15:58:25 +02:00
Aliaksandr Valialkin
c5770600a2
lib/cgroup: follow-up after b9bf3cbe3e
2021-02-08 15:54:38 +02:00
Aliaksandr Valialkin
803a00102a
app/{vminsert,vmselect}: accept requests to paths with /graphite
and /prometheus
prefixes
...
This should improve compatibility with path prefixes from VictoriaMetrics cluster.
See https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#url-format
2021-02-04 20:01:18 +02:00
Aliaksandr Valialkin
b44edc7832
docs/CHANGELOG.md: mention recently added changes
2021-02-04 16:41:34 +02:00
Aliaksandr Valialkin
9fb38569eb
lib/httpserver: expose process_open_fds
and process_max_fds
metrics
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037
2021-02-04 16:40:50 +02:00
Aliaksandr Valialkin
814455138a
docs/CHANGELOG.md: cut v1.53.1
2021-02-03 23:45:21 +02:00
Aliaksandr Valialkin
ea00dac35f
docs/CHANGELOG.md: mention that {__graphite__="foo.*.bar"}
syntax deprecates -search.treatDotsAsIsInRegexps
command-line flag
2021-02-03 20:41:29 +02:00
Aliaksandr Valialkin
a5a1b9bd66
lib/storage: fix a bug, which breaks searching by Graphite wildcard filters
2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
9d41c06db1
docs/CHANGELOG.md: fix a link to Graphite Render API usage docs
2021-02-03 12:29:37 +02:00
Aliaksandr Valialkin
c7e03f30d8
docs: mention about Graphite render API implementation
2021-02-03 12:12:04 +02:00
Aliaksandr Valialkin
8c568b13b2
docs/CHANGELOG.md: cut v1.53.0
2021-02-03 03:42:31 +02:00
Aliaksandr Valialkin
157c02622b
app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"}
syntax
2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
4068f8d590
lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric
2021-02-02 16:15:25 +02:00
Aliaksandr Valialkin
bd11fd8f1d
lib/promscrape: add vm_promscrape_scrape_retries_total
, vm_promscrape_discovery_retries_total
and vm_promscrape_discovery_requests_total
metrics
2021-02-01 20:06:27 +02:00
Aliaksandr Valialkin
b43b498fd8
app/vmselect: add ability to pass extra_label=<label>=<value>
query arg to Prometheus Querying API
...
This enforced `{label="value"}` label filter to the query.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021
2021-02-01 18:04:17 +02:00
Aliaksandr Valialkin
5d87dbfd65
docs: document ability to query Graphite datasource from vmalert
2021-02-01 15:26:33 +02:00
Aliaksandr Valialkin
b2aa80e74b
app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values
...
This commit also adds --vm-round-digits command-line option to vmctl tool.
2021-02-01 14:27:09 +02:00
Aliaksandr Valialkin
d5c180e680
app/vmctl: move vmctl code from github.com/VictoriaMetrics/vmctl
...
It is better developing vmctl tool in VictoriaMetrics repository, so it could be released
together with the rest of vmutils tools such as vmalert, vmagent, vmbackup, vmrestore and vmauth.
2021-02-01 01:10:20 +02:00
Aliaksandr Valialkin
929f09b90d
docs/CHANGELOG.md: typo fixes
2021-01-27 01:18:48 +02:00
Aliaksandr Valialkin
fc5b26d856
lib/promscrape: export vm_promscrape_scrapes_failed_per_url_total
and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total
metrics
...
These metrics could be useful for determining imporperly working scrape targets.
Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.
2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
419ad74269
app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035
2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
3fe848cdd7
lib/logger: add -loggerTimezone
command-line flag for adjusting timezone for timestamps in log messages
2021-01-26 22:51:54 +02:00
Aliaksandr Valialkin
5481906db6
docs/CHANGELOG.md: mention about https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1027
2021-01-26 16:37:36 +02:00
Aliaksandr Valialkin
8cea3c3cc4
lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection
2021-01-22 13:22:33 +02:00
Aliaksandr Valialkin
3caac3d12c
docs/CHANGELOG.md: mention about the fix with too big HTTP reconnection rate to targets
...
This has been fixed in 0a45220b0a
2021-01-22 12:09:16 +02:00
Aliaksandr Valialkin
8749c2dd92
app/vmselect: add -search.maxStepForPointsAdjustment
command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range
handler if they have timestamps closer than -search.latencyOffset
to the current time
2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin
7ce87ebcb2
docs/CHANGELOG.md: cut v1.52.0
2021-01-13 12:58:51 +02:00
Aliaksandr Valialkin
1051d8aa2d
app/vmselect/promql: add ability to pass multiple labels to sort_by_label
and sort_by_label_desc
functions
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin
bdd0a1cdb2
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:16:35 +02:00
Aliaksandr Valialkin
acf1a2c72b
app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
2021-01-13 10:59:42 +02:00
Aliaksandr Valialkin
89315d719d
docs/CHANGELOG.md: document updated extra_label
query arg behavior
...
Follow-up for dc9d7aedd5
2021-01-13 00:58:20 +02:00
Aliaksandr Valialkin
7373986f9e
docs/CHANGELOG.md: mention that the minimum supported TLS version now is v1.2
...
Follow-up for 7bf5d48315
2021-01-13 00:44:39 +02:00
Aliaksandr Valialkin
fe3444b124
deployment/docker: upgrade base image for Docker packages from Alpine 3.13.1 to Alpine 3.12.3 in order to fix potential security issues
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1010
2021-01-12 21:57:01 +02:00
Aliaksandr Valialkin
1837f2f7d3
app/vmselect/promql: add tfirst_over_time(m[d])
and tlast_over_time(m[d])
MetricsQL functions for returning timestamps for the first and the last samples in m
over d
2021-01-12 16:12:12 +02:00
Aliaksandr Valialkin
2140ccbdcc
docs/CHANGELOG.md: document big fixes from the commit 7976c22797
2021-01-12 13:44:17 +02:00
Aliaksandr Valialkin
2c44f9989a
lib/promscrape: properly show scrape duration on /targets
page
...
Previously it has been shown as 0.000s for any scrape duration.
2021-01-11 21:14:46 +02:00
Aliaksandr Valialkin
89611fa48c
docs/CHANGELOG.md: mention about a bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989
2021-01-11 13:11:41 +02:00
Aliaksandr Valialkin
24ffad74c1
all: use net.Dial
instead of fasthttp.Dial
, because fasthttp.Dial
limits the number of concurrent dials to 1000
2021-01-11 12:53:30 +02:00
Aliaksandr Valialkin
9dcb18e03d
app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage
2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin
70df5f4975
docs/CHANGELOG.md: cut v1.51.0
2020-12-27 14:21:29 +02:00
Aliaksandr Valialkin
c86286ec1d
app/vmselect/promql: do not ajdust offset
value provided in the query
...
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin
59183f66d0
app/vmselect: refactor /api/v1/stats/top_queries
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
2020-12-25 16:44:29 +02:00
Aliaksandr Valialkin
932e53522d
docs/CHANGELOG.md: mention that vmalert now properly escapes multi-line queries when passing to Grafana
...
A follow-up for 1de15ad490
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890
2020-12-25 11:12:06 +02:00
Aliaksandr Valialkin
4eb520a342
docs/CHANGELOG.md: mention about adding missing __meta_kubernetes_service_* labels for endpoints
and endpointslices
roles in kubernetes_sd_config
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982
2020-12-24 11:33:00 +02:00
Aliaksandr Valialkin
820669da69
lib/promscrape: code prettifying for 8dd03ecf19
2020-12-24 10:56:10 +02:00
Aliaksandr Valialkin
9df60518bb
docs: mention that it is possible to set multiple -notifier.tlsInsecureSkipVerify
command-line flags for vmalert
...
See c3a92968343c2b3619f1ab935702d0e9b3a46733
2020-12-22 22:32:13 +02:00
Aliaksandr Valialkin
46dba00756
lib/storage: remove stale parts as soon as they go outside the configured retention
...
Previously such parts could remain undeleted for long durations until they are merged with other parts.
This should help for `-retentionPeriod` values smaller than one month.
2020-12-22 19:54:31 +02:00
Aliaksandr Valialkin
e15b20dde3
docs/CHANGELOG.md: cut v1.50.2
2020-12-19 15:32:34 +02:00
Aliaksandr Valialkin
13804bda8f
docs/CHANGELOG.md: mention about bugfix for populating template variables in vmalert
...
See 404cbd1522
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974
2020-12-19 14:16:04 +02:00
Aliaksandr Valialkin
88ac4dfc07
app/vmselect: properly parse negative combined offsets such as -1h2m3s
...
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin
17c2ce18fd
docs/CHANGELOG.md: there is no visible difference for CPU usage and disk IO usage on production workloads
2020-12-19 00:51:41 +02:00
Aliaksandr Valialkin
ebf8da3730
lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage
2020-12-18 20:01:08 +02:00
Aliaksandr Valialkin
e6666da4e7
docs/CHANGELOG.md: mention that Docker images for vmagent, vmalert, vmauth, vmbackup and vmrestore with tags containing -cluster
suffix are no longer published
...
See 441822c4cc
for details
2020-12-18 20:01:03 +02:00
Aliaksandr Valialkin
9abb2d6c74
lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2
...
Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.
2020-12-16 23:40:17 +02:00