1221 Commits

Author	SHA1	Message	Date
weng zhao	cc3e69e963	vmalert: add option datasource.queryStep to allow user to address the inconsistency between grafana dashboards(query_range with step 15s usually) and ALERTS (#1027) ... Co-authored-by: zhao.weng <zhao.weng@shopee.com>	2021-01-26 08:12:04 +00:00
Aliaksandr Valialkin	c164a8d231	app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag ... This should reduce incorrect usage and assumptions for this flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020	2021-01-22 13:00:10 +02:00
Aliaksandr Valialkin	8749c2dd92	app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time	2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin	011c5da785	app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()	2021-01-18 17:30:26 +02:00
Aliaksandr Valialkin	75f7c51cab	docs/vmagent.md: follow-up for 184a659c5f	2021-01-13 13:53:14 +02:00
Aliaksandr Valialkin	1051d8aa2d	app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992	2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin	acf1a2c72b	app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990	2021-01-13 10:59:42 +02:00
Nikolay	dc9d7aedd5	adds extra_label to all import apis (#1007) ... * adds extra_label to all import apis, changes priority for extra_label - now it has priority over original labels * Update README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * Update README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds extra labels to vmagent import api changes order for adding labels, now its added after user values * adds tests for extra_label * import fix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-01-13 00:52:50 +02:00
Aliaksandr Valialkin	1837f2f7d3	app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d	2021-01-12 16:12:12 +02:00
Nikolay	7976c22797	Fixes error handling for promscrape.streamParse (#1009) ... properly return error if client cannot read data, properly suppress scraper errors	2021-01-12 13:31:47 +02:00
Aliaksandr Valialkin	24ffad74c1	all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000	2021-01-11 12:53:30 +02:00
Roman Khavronenko	2e2e4f7e21	vmalert-989: return non-empty result in template func query stub to pass validation (#1002) ... On templates validation stage vmalert does not acutally send queries, so for complex chained expression validation may fail. To avoid this, we add a blank sample in response so validation can pass successfully. Later, during the rule execution, stub will be replaced with real `query` function. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989	2021-01-10 02:56:11 +03:00
Aliaksandr Valialkin	9dcb18e03d	app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage	2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin	c86286ec1d	app/vmselect/promql: do not ajdust offset value provided in the query ... Previously it could be modified in order to improve response cache hit ratio. This is unneeded, since cache hit ratio should remain good because the query time range should be already aligned to multiple of `step` values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976	2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin	4b7105a65b	app/vmselect: sync query stats handling with cluster version	2020-12-27 13:00:29 +02:00
Aliaksandr Valialkin	df0309eae0	app/vmselect/promql: simplify defer call for querystats.RegisterQuery	2020-12-27 12:06:04 +02:00
Aliaksandr Valialkin	ad4e6a9283	app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K ... This should reduce memory usage in constrained environments	2020-12-25 17:40:47 +02:00
Aliaksandr Valialkin	59183f66d0	app/vmselect: refactor /api/v1/stats/top_queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907	2020-12-25 16:44:29 +02:00
Aliaksandr Valialkin	fb338c50a3	app/victoria-metrics: show usage info when incorrect command-line flag is passed to executable	2020-12-25 16:42:21 +02:00
Nikolay	86630350bf	Adds query stats handler (#945) ... * Adds query stat handler, for query and query_range api, victoriametrics tracks query execution time, stats are expored at /api/v1/status/queries endpoint with topN param https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907 * fixed query stats bugs * improves queryStats tracker * improves query stat * small fix * fix tests * added more tests * fixes 386 tests * naming fixes * adds drop for outdated records	2020-12-25 16:42:05 +02:00
Nikolay	1de15ad490	adds escape for CRLF (#984) ... at external.alert.source - \n and \r symbols was url encoded, instead of direct usage. replace it from "\n" to `\n` allows to skip url encoding. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890	2020-12-25 11:03:13 +02:00
Aliaksandr Valialkin	0326638c90	app/vmalert: typo fix in descriptions for notifier.basicAuth.username and notifier.basicAuth.password command-line flags	2020-12-24 12:48:59 +02:00
Aliaksandr Valialkin	9df60518bb	docs: mention that it is possible to set multiple -notifier.tlsInsecureSkipVerify command-line flags for vmalert ... See c3a92968343c2b3619f1ab935702d0e9b3a46733	2020-12-22 22:32:13 +02:00
Nikolay	c270f8f3e6	changes vmalert notifier flag, (#978) ... fixes issue with notifier insecure setting, now its possible to use multiple notifier.tlsInsecureSkipVerify multiple time.	2020-12-22 23:23:04 +03:00
Roman Khavronenko	404cbd1522	vmalert-974: fix order for labels templating (#975) ... The change fixes bug caused by 3adf8c5a6f. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974	2020-12-19 14:10:59 +02:00
Aliaksandr Valialkin	88ac4dfc07	app/vmselect: properly parse negative combined offsets such as -1h2m3s ... Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.	2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin	aa56b9217e	app/vmagent: add vmagent_remotewrite_blocks_sent_total and vmagent_remotewrite_bytes_sent_total metrics per each -remoteWrite.url	2020-12-15 20:39:57 +02:00
Aliaksandr Valialkin	b10ad44692	docs/vmagent.md: typo fix: pearsed->parsed	2020-12-15 19:03:01 +02:00
Aliaksandr Valialkin	1eabbc0e27	docs/vmagent.md: mention that sample_limit option has no sense when stream parsing is enabled	2020-12-15 18:43:41 +02:00
Aliaksandr Valialkin	11fa458e39	app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100	2020-12-15 13:40:46 +02:00
Nikolay	95c9b630cc	adds new Array Flags (#965) ... * adds ArrayDuration and ArrayBool flags, makes sendTimeout and tlsInsecure configurable per remoteWrite url * added backward compatibility testcases for ArrayDuration and ArrayBool * fixes bool flag * fixes test cases	2020-12-15 12:51:12 +02:00
Roman Khavronenko	6247884057	vmalert: add function "query", "first" and "value" to alert templates functions (#960) ... The commit adds a support for template function `query`, `first` and `value`. The function `query` executes a MetricsQL query for active alerts. In vmalert we update templates on every evaluation for active alerts to keep them up to date. With `query` func it may become a perf issue since it will fire a query on every execution. We should keep it in mind for now. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539	2020-12-14 20:11:45 +02:00
Aliaksandr Valialkin	f8e7f433cf	app/victoria-metrics: prettify / page output	2020-12-14 14:07:58 +02:00
Aliaksandr Valialkin	069c9ade52	app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser	2020-12-14 14:06:00 +02:00
Nikolay	ce8c2dd1f1	Changes targets api (#961) ... * changes /targets api adds html response if requester accepts text/html, adds quick template for /targets api, fixes pathPrefix for / requests * changes namings * renamed targets file * Update app/victoria-metrics/main.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds trimspace to qtpl, moves content-type for targets response closer to writer * fixes bug with prefix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-14 13:36:48 +02:00
Aliaksandr Valialkin	5ebfc275e6	app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested	2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin	9d42546a27	docs: consistently use links to https://victoriametrics.github.io for documentation references	2020-12-11 21:08:18 +02:00
Aliaksandr Valialkin	ba8ac08739	app/vmselect/graphite: properly handle wildcards and charsets inside curly braces ... For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952	2020-12-11 17:24:12 +02:00
Aliaksandr Valialkin	4146fc4668	all: properly handle CPU limits set on the host system/container ... This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin	1906f841c9	app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect	2020-12-08 15:16:17 +02:00
Aliaksandr Valialkin	b6b1b06d70	app/{vmalert,vmagent}: skip empty values in -remoteWrite.label and -label lists	2020-12-08 14:55:13 +02:00
Aliaksandr Valialkin	007dbf273d	app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942	2020-12-07 01:08:52 +02:00
Aliaksandr Valialkin	e2e8ef86d9	app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N	2020-12-05 12:30:46 +02:00
Aliaksandr Valialkin	9d787f9edd	all: do not print usage info for all the flags when incorrect command-line flag is passed ... This should improve usability for VictoriaMetrics apps that have big number of command-line flags, i.e. all the apps.	2020-12-03 21:47:37 +02:00
Aliaksandr Valialkin	66379cc69f	app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936	2020-12-03 21:47:36 +02:00
Aliaksandr Valialkin	44a34a0f5f	app/vmselect/promql: make fmt	2020-12-02 21:33:35 +02:00
Aliaksandr Valialkin	1982505c2b	app/vmselect/promql: return nan from minute(m) when m equals to nan ... This aligns VictoriaMetrics behaviour with Prometheus behaviour. The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 20:16:58 +02:00
Aliaksandr Valialkin	9d87496b50	app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. ... This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.	2020-12-02 13:12:50 +02:00
Aliaksandr Valialkin	91a4c279cc	app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true ... This aligns MetricsQL behavior to Prometheus' one. The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 12:09:34 +02:00
Aliaksandr Valialkin	700bda8e2e	app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does	2020-12-02 00:28:26 +02:00
Aliaksandr Valialkin	efdc3c71af	app/vmselect/searchutils: return elapsed time in Deadline.String() output ... This should improve debuggability for error messages containing Deadline.String() output	2020-12-01 00:15:18 +02:00
Aliaksandr Valialkin	ca091bade3	app/vmbackup/snapshot: add missing status code check for the returned response when working with snapshot API ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929	2020-11-30 14:49:39 +02:00
Aliaksandr Valialkin	b35b3dc043	app/vmbackup/snapshot: log url and response body on failed JSON response parsing ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929	2020-11-29 12:16:16 +02:00
Nikolay	0463cb5550	fixes checksum calculation (#928) ... * fixes checksum calculation, 'for' rule param wasnt marshal properly during checksum calculation * fixes error	2020-11-29 09:48:42 +02:00
Aliaksandr Valialkin	b7fcdb528d	app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun	2020-11-25 22:59:13 +02:00
Aliaksandr Valialkin	dabbf930d8	app/vmagent: do not enable -promscrape.config.strictParse when -dryRun command-line flag is set ... Users can specify -promscrape.config.strictParse if -promscrape.config shouldn't contain unknown config entries	2020-11-25 22:26:25 +02:00
BigFish	0acdab3ab9	Update main.go (#922) ... fix spelling mistake	2020-11-23 17:33:17 +02:00
Aliaksandr Valialkin	7e8dcf9ddc	app/vmbackup: cosmetic fixes	2020-11-23 17:10:04 +02:00
Aliaksandr Valialkin	a724dde90a	app/vmselect: protect /tags/delSeries with -deleteAuthKey in the same way as /api/v1/admin/tsdb/delete_series	2020-11-23 15:35:59 +02:00
Aliaksandr Valialkin	f0c207fae2	app/vmselect: add /tags/delSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb	2020-11-23 15:27:21 +02:00
Aliaksandr Valialkin	d3794eb994	app/{vminsert,vmselect}: move /tags/tagSeries and /tags/tagMultiSeries api from vminsert to vmselect ... This is needed for consistency, since all the `/tags*` api handlers are located in vmselect.	2020-11-23 12:33:19 +02:00
Aliaksandr Valialkin	f765985947	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:21 +02:00
Nikolay	eec76718e9	Adds eureka service discovery (#913) ... * Adds eureka service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851 Netflix service discovery for AWS * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-20 13:38:12 +02:00
Aliaksandr Valialkin	2859a452d4	app/vmselect: add remoteAddr to slow query log in order to improve debuggability ... This will simplify identifying the client that sends slow queries to VictoriaMetrics.	2020-11-18 20:38:32 +02:00
Nikolay	f818ab497b	Fixes snap script (#909)	2020-11-18 17:46:31 +03:00
Nikolay	c2afa3fdd7	adds snap package for victoria-metrics (#904) ... * adds snap package for victoria-metrics * Update README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-18 02:00:06 +02:00
Aliaksandr Valialkin	266788be14	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place ... This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin	cf18df367d	app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name	2020-11-16 15:52:16 +02:00
Aliaksandr Valialkin	2c67232565	app/vmselect/graphite: add /tags/autoComplete/values handler from Graphite Tags API	2020-11-16 15:29:35 +02:00
Aliaksandr Valialkin	86f99c6b55	app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support	2020-11-16 14:50:05 +02:00
Aliaksandr Valialkin	3c1434118e	app/vmselect/prometheus: return __name__ label if match[] query to /api/v1/labels matches at least a single time series	2020-11-16 13:54:34 +02:00
Aliaksandr Valialkin	27a417bcd3	app/vmselect/prometheus: improve performance for /api/v1/labels and /api/v1/label/<labelName>/values on time ranges exceeding one day when match[] query arg is set	2020-11-16 13:51:59 +02:00
Aliaksandr Valialkin	6fa806f1ca	app/vmselect/prometheus: fix deadlock in /api/v1/series on a time range exceeding one day	2020-11-16 13:30:47 +02:00
Aliaksandr Valialkin	465923b181	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin	414cd39659	app/vmselect/graphite: apply filter then limit	2020-11-16 04:09:14 +02:00
Aliaksandr Valialkin	d100341394	app/vmselect/graphite: add /tags/<tag_name> handler for Graphite Tags API	2020-11-16 03:42:25 +02:00
Aliaksandr Valialkin	6251762787	app/vmselect/graphite: add /tags handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 03:00:01 +02:00
Aliaksandr Valialkin	48d033a198	app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb	2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin	d9d01f976b	app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points ... Do not take into account spare data point value if the next point will is located too far from the current point. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 15:23:44 +02:00
Aliaksandr Valialkin	1f19c167a4	app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() ... This should remove unexpected spikes at the end of gaps. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 14:59:24 +02:00
Aliaksandr Valialkin	47a038401b	all: consistently return text-based HTTP responses with charset=utf-8 ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:35:41 +02:00
faceair	077f8cbe1c	add charset on targets response (#897)	2020-11-13 10:17:37 +02:00
Aliaksandr Valialkin	4057305148	docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into Relabeling section	2020-11-12 12:27:06 +02:00
Aliaksandr Valialkin	bb06b98202	docs/vmagent.md: typo fix	2020-11-11 16:04:46 +02:00
Aliaksandr Valialkin	4adb96161a	docs/vmagent.md: add Configuration update section	2020-11-11 16:01:15 +02:00
immerrr again	51c529a2b6	app/vmstorage: add "/internal/force_flush" endpoint (#893)	2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin	348edd92fe	app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters	2020-11-11 12:39:07 +02:00
Aliaksandr Valialkin	47390d8947	app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2020-11-10 14:51:44 +02:00
Aliaksandr Valialkin	ba4a2c8bca	app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum	2020-11-10 01:18:08 +02:00
Roman Khavronenko	3adf8c5a6f	vmalert: explicitly set extra labels to alert entities (#886) ... The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, time series always contained only original labels and alert ID was generated from sorted labels key-values. Extra labels didn't affect the generated ID and were applied on the following actions: - templating for Summary and Annotations; - persisting state via remote write; - restoring state via remote read. Such behaviour caused difficulties on restore procedure because extra labels had to be dropped before checking the alert ID, but that not always worked. Consider the case when expression returns the following time series `up{job="foo"}` and rule has extra label `job=bar`. This would mean that restored alert ID will be always different to the real time series because of collision. To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't store original labels anymore. However, this could result into a new error situation. Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`, while rule has extra label `job=bar`. In such case, applying extra labels will result into two identical time series and `vmalert` will return error: `result contains metrics with the same labelset after applying rule labels` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-10 00:27:32 +02:00
Aliaksandr Valialkin	bcd139362b	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 00:19:57 +02:00
Aliaksandr Valialkin	83df20b5b5	lib/promscrape: clean references to label name and label value strings after applying per-target relabeling ... This should reduce memory usage when per-target relabeling creates big number of temporary labels with long names and/or values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 16:19:59 +02:00
Aliaksandr Valialkin	dd6bfa50e9	app/vmselect/promql: code cleanup after 43823addea	2020-11-06 01:30:50 +02:00
n4mine	43823addea	app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883)	2020-11-06 01:29:24 +02:00
Aliaksandr Valialkin	368b69b4c4	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin	b378cd6ed8	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin	381ad564a2	docs/vmagent.md: update after 4c808d58bf	2020-11-04 20:31:43 +02:00
Nikolay	4c808d58bf	Adds ready probe (#874) ... * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:29:18 +02:00
Aliaksandr Valialkin	caeb74f068	app/vmselect: reduce memory usage when query touches big number of time series	2020-11-04 17:04:04 +02:00
Nikolay	e4182dd896	reduces memory usage for vmagent, (#880) ... * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:03:43 +02:00
Aliaksandr Valialkin	600f225cff	Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels" ... This reverts commit bd81f926a4.	2020-11-04 11:44:57 +02:00
Aliaksandr Valialkin	bd81f926a4	docs/vmagent.md: mention about -promscrape.dropOriginalLabels	2020-11-04 11:16:33 +02:00
Aliaksandr Valialkin	e0a91ef163	app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop	2020-11-03 14:25:10 +02:00
Aliaksandr Valialkin	5e75c389e6	app/vmselect/promql: allow dropping trailing sample only for default_rollup function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850	2020-11-02 02:10:59 +02:00
Aliaksandr Valialkin	0b35da159c	app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code ... This makes consistent the behaviour with Prometheus. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873	2020-11-02 00:45:09 +02:00
Aliaksandr Valialkin	cb71af216a	app/vmselect/promql: go fmt	2020-11-02 00:15:29 +02:00
Aliaksandr Valialkin	daacbc7e34	app/vmselect/promql: do not drop trailing datapoints for instant queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-11-02 00:12:37 +02:00
Aliaksandr Valialkin	e277c3d07b	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	b7638f04a7	app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets ... This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	c539494b36	app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K	2020-11-01 23:35:06 +02:00
Roman Khavronenko	f0bdc5716e	vmalert: skip automatically added labels on alerts restore (#871) ... Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correctly purged on restore event and affected alert's ID uniqueness. This commit removes `alertgroup` label in restore function. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-10-30 08:18:20 +00:00
kreedom	a51a7b2a20	vmbackup fix panic when no origin fs given (#859) ... * use fsnil when no origin fs	2020-10-28 20:09:10 +02:00
Aliaksandr Valialkin	5bfd4e6218	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:31:44 +03:00
kreedom	ef77120170	vmalert - add dryRun (#842) ... vmalert: add `dryRun` flag for rules validation without running the service	2020-10-20 08:15:21 +01:00
Seva Poliakov	b3f3c078e5	Fix typo in vnrestore readme	2020-10-18 15:41:39 +03:00
Aliaksandr Valialkin	28353e48ca	app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series ... Now trailing data points are additionally dropped for time series with a single raw sample Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-17 10:44:34 +03:00
Roman Khavronenko	bc42b5598f	vmalert: update docs to highlight the state restore requirements; (#833) ... Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830	2020-10-13 18:32:43 +03:00
Aliaksandr Valialkin	a2b9476897	app/vmselect/promql: return a single time series at max from absent() function like Prometheus does	2020-10-13 15:56:04 +03:00
Aliaksandr Valialkin	9aa3b65766	app/vmselect/promql: improve time series staleness detection ... This should prevent from double counting for time series at the time when it changes label. The most common case is in K8S, which changes pod uid label with each new deployment. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-13 12:19:57 +03:00
Aliaksandr Valialkin	d8af290947	app/vmselect/promql: fix mode_over_time calculations ... Previously `mode_over_time` could return garbage due to improper shuffling of input data points.	2020-10-13 11:58:25 +03:00
Aliaksandr Valialkin	1e27420243	app/vmselect/prometheus: fix golangci-lint warning	2020-10-13 09:36:11 +03:00
Aliaksandr Valialkin	4f16a964e3	app/vmselect: add ability to export data in CSV format via /api/v1/export/csv	2020-10-12 20:08:17 +03:00
Aliaksandr Valialkin	63c4999e06	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:12:36 +03:00
Nikolay Khramchikhin	9bd9f67718	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 13:38:21 +03:00
Aliaksandr Valialkin	762c967855	app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning ... Functions are: * keep_last_value * keep_next_value * interpolate * running_min * running_max * running_avg * range_min * range_max * range_avg * range_first * range_last * range_quantile * smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:47:06 +03:00
Aliaksandr Valialkin	45f7cdc532	Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" ... This reverts commit ac45082216. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric names: * ceil * floor * round Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:40:34 +03:00
Aliaksandr Valialkin	a94825b169	Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" ... This reverts commit bb61a4769b. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * clamp_min * clamp_max Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:38:27 +03:00
Aliaksandr Valialkin	f7d28bddbf	Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" ... This reverts commit e5202a4eae. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * max_over_time * min_over_time * avg_over_time * quantile_over_time * geomean_over_time * mode_over_time * holt_winters * predict_linear Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:35:18 +03:00
Aliaksandr Valialkin	b449607181	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin	cf5f2874cd	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:12:03 +03:00
Aliaksandr Valialkin	68f0e00761	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin	84227ea2fc	app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics	2020-10-09 13:29:51 +03:00
Aliaksandr Valialkin	f4e8687c88	app/vmalert: accept days, weeks and years in for: part of config like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817	2020-10-08 20:13:15 +03:00
Aliaksandr Valialkin	6105d61d11	docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors command-line flag usage	2020-10-08 19:24:31 +03:00
Aliaksandr Valialkin	71ea4935de	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:58:30 +03:00
Aliaksandr Valialkin	d423d73251	app/vmalert: do not pring description for all the flags on config errors ... The description is too big to consume by human and it just distracts humans.	2020-10-08 13:35:57 +03:00
Aliaksandr Valialkin	e9f2e2cbc9	app/vmselect/promql: add missing label filters to binary operands before query execution ... This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization See also https://github.com/cortexproject/cortex/issues/3253	2020-10-07 21:15:09 +03:00
Dmitry Shihovtsev	92e5d89fc9	Fix typos in the vmalert datasource (#814) ... * Fix typos in the vmalert datasource * Fix typo in the vmalert datasource test	2020-10-07 17:59:50 +03:00
Aliaksandr Valialkin	af90b3121c	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:35:44 +03:00
Aliaksandr Valialkin	d16dbfd639	app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page	2020-10-06 15:29:49 +03:00
Aliaksandr Valialkin	cfd720e772	app/victoria-metrics: add a link to https://victoriametrics.github.io/ from main page	2020-10-06 15:29:49 +03:00
Nikolay Khramchikhin	cbe3cf683b	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:45:33 +03:00
Aliaksandr Valialkin	59c26feefa	app/vmselect/promql: fill gaps on graphs for range_* and running_* functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806	2020-10-02 13:59:45 +03:00
Aliaksandr Valialkin	764dc2499f	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:06:23 +03:00
Roman Khavronenko	daa2d1c065	vmalert: make maxIdleConnections configurable for datasource HTTP client (#797) ... Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795	2020-09-30 09:49:45 +03:00
Aliaksandr Valialkin	44bcda81ab	app/vmstorage: rename vm_{big|small}_merge_need_free_disk_space to vm_merge_need_free_disk_space ... This simplifies alerting.	2020-09-29 22:44:19 +03:00
Aliaksandr Valialkin	a9db81c4ab	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:48:33 +03:00
Aliaksandr Valialkin	127537d631	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:12 +03:00
Aliaksandr Valialkin	f7636b0342	app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output	2020-09-29 18:03:26 +03:00
Aliaksandr Valialkin	7dc67cd883	docs/{vmbackup,vmrestore}: formatting fixes	2020-09-29 13:19:07 +03:00
Aliaksandr Valialkin	efdefbc1cb	docs/vmbackup.md: make docs about minio config more prominent	2020-09-29 13:16:04 +03:00
Aliaksandr Valialkin	9945b8c98d	docs: improve readability a bit	2020-09-29 13:03:38 +03:00
Nikolay Khramchikhin	1e679f3e0d	update vmbackup/vmrestore README usage (#794) ... * update vmbackup/vmrestore README usage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/381 with minio and configuration file examples. * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * added backup/restore docs changes * added example for relabelConfig flag * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-29 12:53:10 +03:00
Aliaksandr Valialkin	96ee276e6e	app/vmselect/prometheus: check for errors returned from bufferedwriter.Write ... This makes `make errcheck` happy	2020-09-29 11:37:01 +03:00
Aliaksandr Valialkin	6fdfc67620	app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.* ... In this case only node must be returned with stripped dot in the end of id as carbonapi does	2020-09-29 11:01:59 +03:00
Aliaksandr Valialkin	124f78857b	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin	1b3efccb24	app/vmselect: stop /api/v1/export/* execution if client disconnects	2020-09-27 23:53:13 +03:00
Aliaksandr Valialkin	95688cbfc5	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin	c00627c103	app/vminsert: code prettifying	2020-09-26 04:13:18 +03:00
Aliaksandr Valialkin	b6a976b98d	app/vmagent: reduce memory usage when importing data via /api/v1/import ... Previously vmagent could use big amounts of RAM when each ingested JSON line contained many samples.	2020-09-26 04:10:24 +03:00
Aliaksandr Valialkin	82973f8ae7	Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" ... This reverts commit bab6a15ae0. Reason for revert: the `fetchData` arg is used in cluster branch. Leaving this arg in master branch makes smaller the diff with cluster branch.	2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin	bab6a15ae0	lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock ... This arg became unused after 23bdc1f107	2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin	23bdc1f107	app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls	2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin	c584aece38	app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784	2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin	2985077c35	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:46:34 +03:00
Aliaksandr Valialkin	27500d7d4c	app/vmselect/prometheus: code cleanup after 3ba507000c	2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin	3ba507000c	app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin	bed25e3c24	app/vmselect/netstorage: properly pre-allocate space for sbs	2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin	09b0f7c202	app/vmselect/netstorage: release search resources on timeout errors ... Previously these resources weren't released, which could lead to resource leaks.	2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin	3b1e3a03e0	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin	94f7d00537	docs/vmagent.md: typo fix	2020-09-21 21:49:22 +03:00
Aliaksandr Valialkin	00b5145c69	app/vmselect/searchutils: fixed tests after 2eb72e09ab	2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin	2eb72e09ab	app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:40 +03:00
Nikolay Khramchikhin	312fead9a2	Add improvements to ec2_sd_discovery (#775) ... * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin	1e1a27d803	app/vmalert: remove unneeded UTC() call ... UTC() doesn't change the underlying timestamp, so the call isn't needed here	2020-09-21 15:55:59 +03:00
Roman Khavronenko	5dffc7a553	vmalert: add support for datasource.lookback flag (#779) ... New datasource flag `datasource.lookback` defines how far to look into past when evaluating queries. Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668	2020-09-21 15:53:49 +03:00
Roman Khavronenko	82c3bbce34	vmalert: fix the typo in error message (#782) ... The error will be always nil so no sense in printing it.	2020-09-21 11:34:23 +03:00
Aliaksandr Valialkin	d50165ad59	app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit	2020-09-18 14:21:54 +03:00
Aliaksandr Valialkin	98d1cd0971	app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API	2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin	7a134b0fd7	app/vmstorage: added -forceMergeAuthKey command-line flag for protecting /internal/force_merge endpoint	2020-09-17 14:21:53 +03:00
Aliaksandr Valialkin	1f33dd717f	lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions ... This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin	ab53cb6f7b	app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens ... Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin	9f79bcf64a	app/vmselect: improve description for -search.maxQueryDuration	2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin	1fec47a289	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin	8c3d7c1a59	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin	0e533d1a9c	app/vmselect/promql: support composite durations like Prometheus 2.21 does ... The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 23:39:13 +03:00
Roman Khavronenko	6ad6480400	vmalert: add Group name as label to generated alerts and timeseries (#761) ... Solves #611	2020-09-11 20:52:56 +01:00
Roman Khavronenko	4cdffb04a4	vmalert: update groups on config reload only if changes detected (#759) ... On config reload event `vmalert` reloads configuration for every group. While it works for simple configurations, the more complex and heavy installations may suffer from frequent config reloads. The change introduces the `checksum` field for every group and is set to md5 hash of yaml configuration. The checksum will change if on any change to group definition like rules order or annotation change. Comparing the `checksum` field on config reload event helps to detect if group should be updated. The groups update is now done concurrently, so reload duration will be limited by the slowest group now. Partially solves #691 by improving config reload speed.	2020-09-11 20:14:30 +01:00
Aliaksandr Valialkin	ca856284e4	app/vmagent: allow setting multiple identical -remoteWrite.url values ... This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755	2020-09-11 15:17:22 +03:00
Aliaksandr Valialkin	a2f647d142	app/vmselect/prometheus: typo fix in the description for -search.latencyOffset command-line flag	2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin	2380e9b017	app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:27:14 +03:00
Aliaksandr Valialkin	f0005c3007	app/vmselect: move Deadline from netstorage to searchutils ... This removes dependency on netstorage from searchutils.	2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin	2114179e19	app/vmselect: substitute inf values at smooth_exponential with the previous values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin	204ec415b4	app/vmselect: skip infinite values when calculating smooth_exponential ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin	8a8b5a73d3	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin	f6bc608e86	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin	9d8fdff6c5	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps ... This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin	d7c04db1fc	docs: sync docs for vmalert, vmauth, vmbackup and vmrestore	2020-09-09 21:10:34 +03:00
Aliaksandr Valialkin	11eaa37111	docs/vmagent.md: clarified the case when -remoteWrite.queues must be tuned ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745	2020-09-08 20:15:27 +03:00
Aliaksandr Valialkin	62919eaf7e	app/vmselect/promql: go fmt	2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin	e6da63dffe	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin	8e85b56737	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin	c0343a661b	app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3	2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin	f4e7e5fb90	app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions ... These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin	e706e59d49	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin	ddabc13796	app/vmagent: properly flush big blocks of data ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741 Thanks to @IceRain00 for the investigation and initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742	2020-09-03 12:12:39 +03:00
Aliaksandr Valialkin	7a839b461f	app/vmagent: fix data race when accessing writeRequest.lastFlushTime	2020-09-03 12:12:37 +03:00
Nikolay Khramchikhin	764b3d4fda	changed vmalert behaviour (#738) ... * VMAlert start with empty rules dir There are some applications (operator for instance), that generates alerts configuration at runtime and vmalert must start correctly without rules to support this behaviour. Later application will add rules files and send SIGHUP to vmalert, which will trigger reading rules files and start rules exectuion. Removing rules files with SIGHUP signal must stop rules execution and vmalert will wait for new rules. * imports sorted * added test cases for empty rules, removed blank line * fixed imports conflict * updated tests	2020-09-03 11:04:42 +03:00
Aliaksandr Valialkin	5f16ceb294	app/vmalert: imrovements over 3f932c2db1	2020-09-03 01:00:55 +03:00
DexterZhang	3f932c2db1	feat: spread load of rule evaluation by group when starting new groups (#724) ... * feat: spread load of rule evaluation by group when starting new groups * review: reduce the resulting diff. * Update app/vmalert/group.go Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-03 00:58:54 +03:00
Aliaksandr Valialkin	f41b36bb9a	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats ... Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:43:21 +03:00
Aliaksandr Valialkin	5af777469a	app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url	2020-08-30 21:40:22 +03:00
Aliaksandr Valialkin	2149733bd2	app/vmagent: apply sane limits to -remoteWrite.queues ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707	2020-08-30 21:25:37 +03:00
Roman Khavronenko	10601bc652	vmalert: update -rule flag description to enforce quotes using (#709) ... Description for `-rule` flag uses as example specific chars like asterisks which could be interpreted wrong by different shells. To avoid this, description now contains quoted flag values. See also #708	2020-08-20 22:36:38 +01:00
Roman Khavronenko	f5f59896ec	lib/decimal: rename significant decimal digits to significant figures (#698) ... The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:21:35 +03:00
Aliaksandr Valialkin	147c35ebd4	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin	7d89fafe1a	app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)	2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin	cd96248480	docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion	2020-08-14 20:47:57 +03:00
Aliaksandr Valialkin	184670fb9b	docs: update docs	2020-08-14 19:13:42 +03:00
Aliaksandr Valialkin	576da0fe46	app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag	2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin	c402265e88	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components ... Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:15:25 +03:00
Aliaksandr Valialkin	ef7e2af8f5	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 22:59:19 +03:00
Aliaksandr Valialkin	e969ef2639	app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin	8adba82c02	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores ... This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin	f3d33e23c9	app/vmstorage: improve error logging when the request times out	2020-08-10 13:23:26 +03:00
Roman Khavronenko	0be5b09fb4	app/vmalert: extend metrics set exported by vmalert #573 (#654) ... * app/vmalert: extend metrics set exported by `vmalert` #573 New metrics were added to improve observability: + vmalert_alerts_pending{alertname, group} - number of pending alerts per group per alert; + vmalert_alerts_acitve{alertname, group} - number of active alerts per group per alert; + vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error during prev execution, is 0 if no errors happened; + vmalert_recording_rules_error{recording, group} - is 1 if recording rule ended up with error during prev execution, is 0 if no errors happened; * vmalert_iteration_total{group, file} - now contains group and file name labels. This should improve control over specific groups; * vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups; Some collisions for alerts and recording rules are possible, because neither group name nor alert/recording rule name are unique for compatibility reasons. Commit contains list of TODOs for Unregistering metrics since groups and rules are ephemeral and could be removed without application restart. In order to unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13 * app/vmalert: extend metrics set exported by `vmalert` #573 The changes are following: * add an ID label to rules metrics, since `name` collisions within one group is a common case - see the k8s example alerts; * supports metrics unregistering on rule updates. Consider the case when one rule was added or removed from the group, or the whole group was added or removed. The change depends on https://github.com/VictoriaMetrics/metrics/pull/16 where race condition for Unregister method was fixed.	2020-08-09 09:41:29 +03:00
ofen	81746d14b9	401 Unauthorize HTTP error added (#681) ... 401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]	2020-08-09 09:38:41 +03:00
Aliaksandr Valialkin	716754fae6	app/vmselect/promql: properly handle -n^m like Prometheus does ... `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin	bb61a4769b	app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin	ac45082216	app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin	e5202a4eae	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus ... Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin	68e4f40a72	app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does ... This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin	bc8381613d	app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names	2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin	3f85c06b65	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block ... This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin	d20c2156e4	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin	dbbdfbe7ee	app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"} to vm_cache_entries{type="storage/prefetchedMetricIDs"} to be consistent with other vm_cache_entries metrics	2020-08-06 16:34:24 +03:00
Aliaksandr Valialkin	8f16388428	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 ... This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin	c25b0c2cd5	app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663	2020-08-04 21:00:29 +03:00
Aliaksandr Valialkin	5d0c37bec0	app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration	2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin	e6eee2bebf	app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...)	2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin	509d12643b	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin	d01f3c1943	all: add mssing APP_NAME to vm*-GOARCH builds	2020-07-31 13:42:18 +03:00
Aliaksandr Valialkin	3f498cf2dc	docs/{vmagent,vmalert}: add instruction on how to build for ARM	2020-07-31 09:25:22 +03:00
Aliaksandr Valialkin	f0c678c41b	app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin	e7959094f6	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected ... The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin	328b52e5ff	app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window ... Just divide the data point value by the window duration in this case.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	700737c181	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now ... See e00cfc854d for the approach used now.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	2f735f112d	app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	0f63da3698	app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query ... Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:25:19 +03:00
Aliaksandr Valialkin	62ed38c6f0	app/vmagent/remotewrite: add missing resp.Body.Close() after pushing data to remote storage ... Missing body close could disable HTTP keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:00:15 +03:00
Aliaksandr Valialkin	79c30cf4cb	app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query	2020-07-28 15:13:08 +03:00
Roman Khavronenko	2f1e7298ce	app/vmalert: support external.label to specify global labelset for all rules #622 (#652) ... `external.label` flag supposed to help to distinguish alert or recording rules source in situations when more than one `vmalert` runs for the same datasource or AlertManager.	2020-07-28 14:20:31 +03:00
Aliaksandr Valialkin	0da202023b	app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp ... This aligns `group()` behaviour to Prometheus	2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin	cb8c6908dc	app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage ... Previously the request body was already consumed before the retry, so this led to the following error: http: ContentLength=... with Body length 0	2020-07-27 17:32:46 +03:00
Aliaksandr Valialkin	894dcb7c1c	app/vmselect/promql: improve further the accuracy of buckets_limit() function ... The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin	215eba0b82	app/vminsert: flush bufs if needed after the current row is added ... Previously the data for the added row could be overwritten by the flush before the row addition is complete.	2020-07-26 12:10:11 +03:00
Aliaksandr Valialkin	edb1eca6f1	app/vmselect/promql: avoid dropping inf bucket in buckets_limit ... The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin	97b6f5d223	app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets	2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin	a090627059	app/vminsert: limit memory usage when ingesting data in big packets	2020-07-24 23:32:40 +03:00
Aliaksandr Valialkin	bb161497cf	app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function ... Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin	994fa2f3bf	app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k ... This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin	e151c5c644	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin	3107c633e3	app/vmselect/promql: typo fix after 3e557c9861	2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin	3e557c9861	app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration ... Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin	b1f6843bd0	app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time ... This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin	039c9d2441	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin	2a45871823	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin	461481fbdf	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery	2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin	4c8b49b193	app/vminsert: export vm_relabel_metrics_dropped_total metric that shows the number of metrics dropped due to relabeling	2020-07-23 14:57:53 +03:00
Aliaksandr Valialkin	e79de9774b	app/vmselect: typo fix after 34563916f7	2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin	34563916f7	app/vmselect: reduce memory usage when querying big number of time series with long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin	9257eee982	app/vminsert: do not call ApplyRelabeling function if relabeling is disabled ... This should reduce CPU usage a bit when `-relabelConfig` isn't set	2020-07-23 13:39:44 +03:00
Aliaksandr Valialkin	6f05c4d351	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin	2f612e0c67	app/vminsert: fix relabeling for metrics ingested via Influx line protocol ... Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels if a single Influx line protocol message contains multiple field values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638	2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin	4f526cc816	app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin	dfb113f175	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries ... This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.	2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin	31ae5911a8	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin	caa2952aa6	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin	e00cfc854d	app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value ... This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin	b9c8f6bf34	app/vmselect: log the total available memory for concurrent requests on not enough memory errors ... This should simplify root cause analysis	2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin	ad6290953c	app/vmagent: add -remoteWrite.proxyURL command-line option ... This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy. This is similar to `proxy_url` option in `remote_write` section of Prometheus. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write	2020-07-20 19:28:49 +03:00
Aliaksandr Valialkin	efcbb51968	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-20 17:08:34 +03:00
Roman Khavronenko	ed0df37ee7	app/vmagent: mention grafana dashboard in README (#639)	2020-07-20 17:07:27 +03:00
Aliaksandr Valialkin	11be704109	app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries	2020-07-20 15:49:27 +03:00
Aliaksandr Valialkin	5a4675c528	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range ... This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin	ecb1b2564a	app/vmselect/promql: add mode() aggregate function	2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin	b35cb293f5	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin	1c641037e8	app/vmselect/promql: add mode_over_time(m[d]) function ... See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin	6b5ad535ae	app/vmselect/promql: optimize group(rollup(m)) calculations	2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin	8949d65ad1	app/vmselect/promql: check that any() doesn't touch metric name	2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin	aa5d88055d	app/vmselect/promql: add group() aggregate function to MetricsQL ... This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin	df01836818	app/vmselect/promql: keep all labels for time series from any() call	2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin	8c14ca93fa	app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call ... Previously this could lead to `out of range` panic	2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin	e4e1cd1de2	app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage	2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin	ed7580ad22	app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf when wrapping errors	2020-07-15 13:56:47 +03:00
Aliaksandr Valialkin	328814ee60	docs/vmagent.md: make filtering rules for init container pods less confusing	2020-07-14 20:32:47 +03:00
Aliaksandr Valialkin	b442a42d8e	app/vmagent/remotewrite: return proper value from tssRelabelPool.New ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-14 14:29:20 +03:00
Aliaksandr Valialkin	4081e2295e	app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626	2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin	25f80d320b	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time ... Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin	55d83e777d	app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9	2020-07-13 21:40:38 +03:00
faceair	1033dc7e2a	fix empty response template (#617)	2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin	619b0a25c9	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-13 21:25:11 +03:00
ofen	666c795b98	Update README.md (#621) ... Troubleshooting section updated to help out with duplicate targets detection	2020-07-13 21:18:54 +03:00
Aliaksandr Valialkin	a730b3f6a1	app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig options are set ... Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race and improper relabeling. Now each goroutine has its own copy of tss during relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-10 15:16:59 +03:00
Aliaksandr Valialkin	508ad46e0e	app/vmagent/remotewrite: typo fix in -remoteWrite.showURL help message	2020-07-10 14:07:08 +03:00
Aliaksandr Valialkin	cba820e390	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin	9c350bc20d	app/vmselect/promql: add missing tests for ifnot binary operation	2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin	256fd9a87e	app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation	2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin	2d9b3ad5b3	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin	742da690f4	app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575	2020-07-08 18:55:38 +03:00
DexterZhang	99f54e44ff	feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598)	2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin	6daa5f7500	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:42:05 +03:00
Roman Khavronenko	703def4b2e	app/vmalert: add retries to remotewrite (#605) ... * app/vmalert: add retries to remotewrite Remotewrite pkg now does limited number of retries if write request failed. This suppose to make vmalert state persisting more reliable. New metrics were added to remotewrite in order to track rows/bytes sent/dropped. defaultFlushInterval was increased from 1s to 5s for sanity reasons. * fix * wip * wip * wip * fix bits alignment bug for 32-bit systems * fix mistakenly dropped field	2020-07-05 18:46:52 +03:00
Aliaksandr Valialkin	de137aef98	app/victoria-metrics: fix tests after the commit acf828a759	2020-07-05 18:24:41 +03:00
Aliaksandr Valialkin	acf828a759	app/vmselect/prometheus: small fixes on top of 8bb762124a	2020-07-05 18:17:06 +03:00
faceair	8bb762124a	fix adjust last points avoid influence earlier value (#606)	2020-07-05 17:56:54 +03:00
Ween	d59cdbe90c	[VMAlert] Fix error log when remoteWrite queue size is full (#602) ... * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors * fix error logs when queue is full Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-07-03 16:49:37 +03:00
Aliaksandr Valialkin	0b2086b7a5	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin	8f628cd805	app/victoria-metrics: removed debug log message when -selfScrapeInterval is set	2020-07-02 20:39:41 +03:00
Aliaksandr Valialkin	91b3482894	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin	e5500bfcf2	all: typo fix: exptected -> expected	2020-07-02 18:05:52 +03:00
Aliaksandr Valialkin	5d3db3ff7c	app/vmselect: add interpolate function for filling gaps with linearly interpolated values ... See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin	4dd3de9286	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:14 +03:00
BigFish	9d5f5b6878	fix: spelling mistakes (#594) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-07-01 01:35:26 +03:00
Aliaksandr Valialkin	84a37098ed	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention ... VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention if `-denyQueriesOutsideRetention` command-line flag is set.	2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin	7c2c8b2981	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Roman Khavronenko	88538df267	app/vmalert: support multiple notifier urls (#584) (#590) ... * app/vmalert: support multiple notifier urls (#584) User now can set multiple notifier URLs in the same fashion as for other vmutils (e.g. vmagent). The same is correct for TLS setting for every configured URL. Alerts sending is done in sequential way for respecting the specified URLs order. * app/vmalert: add basicAuth support for notifier client (#585) The change adds possibility to set basicAuth creds for notifier client in the same fasion as for remote write/read and datasource.	2020-06-29 22:21:03 +03:00
Roman Khavronenko	82ecfa3b32	app/vmalert: move flags description and initialization into subpackages ... The change adds no new functionality and aims to move flags definitions to subpackages that are using them. This should improve readability of the main function.	2020-06-28 12:26:22 +01:00
kreedom	dc4e3f0e0b	app/vmalert: properly set transport for HTTP clients ... Fixes issue #586	2020-06-27 08:31:54 +01:00
Aliaksandr Valialkin	8f2e88234f	docs: update the info that docker images are built on top of alpine image now ... A follow-up after the commit ff624c9125 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-26 13:54:10 +03:00
Aliaksandr Valialkin	05d754d7bb	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series ... This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:38:39 +03:00
nicbaz	774f7ca1c1	vmselect: fix label_replace when mismatch (#579) ... As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the **MetricsQL** `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin	7209d58fbd	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series ... Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:46:15 +03:00
nicbaz	72c90bfd8b	vmalert: add support for TLS configuration (#578) ... app/vmalert: add support for TLS configuration Add support for TLS optional configuration in a similar fashion to what is currently supported in other vmutils such as vmagent. TLS configuration options are distinct for datasource, remoteRead, remoteWrite as well as notifier.	2020-06-23 20:45:45 +01:00
Aliaksandr Valialkin	2a39ba639d	lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs ... These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin	7532dbcdf5	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:33:01 +03:00
kreedom	7ec6711f06	Support of custom URL path for alert (#560) ... app/vmalert: Support custom URL for alerts source Add flag `external.alert.source` for configuring custom URL for alert's source. This may be handy to re-point default source URL to other systems like Grafana. Updates #517	2020-06-21 11:32:46 +01:00
Aliaksandr Valialkin	9f3e3a4d7a	docs/vmauth.md: mention that we can provide custom integration with SAML	2020-06-19 13:14:07 +03:00
Aliaksandr Valialkin	b542e50680	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin	88e1b7d144	app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin	a12364ad37	docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible	2020-06-18 18:52:43 +03:00
Roman Khavronenko	e91d758831	vmalert-537: allow name duplication for rules within one group. (#559) ... Uniqueness of rule is now defined by combination of its name, expression and labels. The hash of the combination is now used as rule ID and identifies rule within the group. Set of rules from coreos/kube-prometheus was added for testing purposes to verify compatibility. The check also showed that `vmalert` doesn't support `query` template function that was mentioned as limitation in README.	2020-06-15 20:15:47 +01:00
Clémence Saussez	e426434770	app/vmalert: fix link to testdata (#547) ... Fix broken link to vmalert test data Signed-off-by: Clemence Saussez <clemence@zen.ly>	2020-06-10 17:26:27 +01:00
Roman Khavronenko	3e277020a5	vmalert-491: allow to configure concurrent rules execution per group. (#542) ... The feature allows to speed up group rules execution by executing them concurrently. Change also contains README changes to reflect configuration details.	2020-06-09 15:21:20 +03:00
Roman Khavronenko	ffa75c423d	vmalert-521: allow to disable rules expression validation. (#536) ... This feature may be useful for using `vmalert` with PromQL compatible datasources like Loki.	2020-06-06 21:27:09 +01:00
Aliaksandr Valialkin	f5dd2a71a6	app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 20:13:56 +03:00
Aliaksandr Valialkin	4b98e436ef	app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set	2020-06-05 20:13:55 +03:00
Aliaksandr Valialkin	0d92abfbf6	app/vmalert: print brief usage info for vmalert -help	2020-06-05 10:43:18 +03:00
Aliaksandr Valialkin	ff1a725a56	app/vmauth: print brief usage info for vmauth -help	2020-06-05 10:40:00 +03:00
Aliaksandr Valialkin	05ae1472e3	app/vmagent: print brief usage info for vmagent -help	2020-06-05 10:39:59 +03:00
Aliaksandr Valialkin	f8692a1d43	app/vmauth: log when -auth.config is reloaded in SIGHUP	2020-06-03 23:22:14 +03:00
Aliaksandr Valialkin	d2f30e8d79	app/vmalert: fix comment for UpdateWith exported methods	2020-06-01 14:35:32 +03:00
Roman Khavronenko	270552fde4	vmalert: Add recording rules support. (#519) ... * vmalert: Add recording rules support. Recording rules support required additional service refactoring since it wasn't planned to support them from the very beginning. The list of changes is following: * new entity RecordingRule was added for writing results of MetricsQL expressions into remote storage; * interface Rule now unites both recording and alerting rules; * configuration parser was moved to separate package and now performs more strict validation; * new endpoint for listing all groups and rules in json format was added; * evaluation interval may be set to every particular group; * vmalert: uncomment tests * vmalert: rm outdated TODO * vmalert: fix typos in README	2020-06-01 13:46:37 +03:00
Aliaksandr Valialkin	32652485e3	app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518	2020-05-30 14:37:12 +03:00
Aliaksandr Valialkin	d988f89415	app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent	2020-05-30 14:23:14 +03:00
Aliaksandr Valialkin	5b6a9675d8	app/vmauth: fix make run-vmauth command	2020-05-22 16:45:02 +03:00
Aliaksandr Valialkin	8905bc2a40	app/vmagent: check for error returned from flag.Set	2020-05-21 16:31:14 +03:00
Aliaksandr Valialkin	f9847352b4	app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin	dbd8beccfa	app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions ... These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:07:48 +03:00
kreedom	6b23df2bec	vmalert add quotes escape function (#510) ... * vmalert add quotes escape function Co-authored-by: kreedom	2020-05-20 22:20:31 +03:00
Aaron France	619d4959c7	Update README.md	2020-05-20 09:04:31 +03:00
Aliaksandr Valialkin	70ea4e28a7	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin	80c18b7275	docs/vmagent.md: mention an alternative to refresh_interval option in scrape configs	2020-05-19 23:10:06 +03:00
Aliaksandr Valialkin	538fdfe133	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin	f52769f6ee	app/vmselect/promql: fix outilersk calculations	2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin	a441cdd1d9	app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin	d0f08b4a58	app/vmalert/notifier: go fmt	2020-05-19 12:59:46 +03:00
kreedom	7e173655ba	vmalert - add expr to variables, add escape functions (#495) ... * vmalert - add expr to variables, add escape functions Co-authored-by: kreedom	2020-05-18 11:55:16 +03:00
Roman Khavronenko	92212f04da	vmalert: avoid sending resolves for pending alerts (#498) ... Before the change we were sending notifications to notifier if following conditions are met: * alert is in Fire state * alert is in Inactive state We were sending Inactive notifications to resolve alert ASAP. Unfortunately, we were sending resolves for Pending alerts that become Inactive, which is wrong. In this change we delete alert from the active list if it was Pending and become Inactive. In this way we now have Inactive alerts only if they were in state Fire before. See test change for example.	2020-05-17 15:13:22 +01:00
Roman Khavronenko	de60ad0cd6	vmalert: fix potential race during configuration reloads (#497) ... Configuration reload and rules evaluation can't be executed in same time now. This may make reload time longer but prevents from potential races.	2020-05-17 15:12:09 +01:00
Aliaksandr Valialkin	eac3da478e	app/vmalert: run make quicktemplate-gen from the root dir of the repository	2020-05-16 22:46:02 +03:00
Aliaksandr Valialkin	93c87d28f6	all: print --help output to stdout instead of stderr ... This is easier to grep and pipe	2020-05-16 11:59:33 +03:00
Aliaksandr Valialkin	8cb35974af	app/vmrestore: document better that vmrestore works like rsync --delete, i.e. it deletes files in -storageDataPath, which are missing in the backup	2020-05-16 09:22:17 +03:00
Aliaksandr Valialkin	3a68c47de0	app/vmagent/Makefile: fix make run-vmagent rule	2020-05-15 19:35:10 +03:00
Aliaksandr Valialkin	697b6af10f	app/vmagent/remotewrite: remove unused import after the commit 93267f143f	2020-05-15 17:42:19 +03:00
Aliaksandr Valialkin	93267f143f	app/vmagent/remotewrite: allow ingesting time series with multiple samples at once ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481	2020-05-15 17:36:36 +03:00
Aliaksandr Valialkin	82ffbcb9a6	app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin	82ccdfaa91	app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:44:32 +03:00
Roman Khavronenko	a249cd9d22	vmalert: fix the access to rules slice element by wrong index (#486) ... During group's update rules deletion was causing slice mutations while slice index was assumed to be unchanged. This caused "slice bounds out of range" errors when multiple rules were deleted sequentially.	2020-05-15 07:55:22 +01:00
hagen1778	ef0e37cb9e	vmalert: update README	2020-05-15 09:17:28 +03:00
Aliaksandr Valialkin	0afd48d2ee	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin	894e5d2b9b	docs/vmbackup.md: add a link to vmbackuper tool	2020-05-13 22:54:22 +03:00
Roman Khavronenko	415b1ddfb5	vmalert: check if remoteRead object was initied before calling Restore (#473) ... The check for non-nil remoteRead was mistakenly dropped during refactoring which caused panics when `vmalert` wasn't configured with `remoteRead` flag.	2020-05-13 19:32:58 +01:00
Roman Khavronenko	db7dd96346	vmalert: fix flag names and description in README (#475) ... Change also adds the recommendation for `remotewrite` queue error.	2020-05-13 19:32:21 +01:00
肖贝贝	ba48438b06	Feat/vmalert add max queue size (#472) ... * feat: add remoteWrite.maxQueueSize to reduce queue full * rename remote(write|read) flags to remote(Write|Read) for the sake of consistency Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-13 18:58:56 +01:00
Aliaksandr Valialkin	7882a0dbbf	app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint	2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin	96e001d254	app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set ... This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin	faf92a0965	app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one	2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin	cc311e20fe	app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y	2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin	574289c3fb	app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation	2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin	0a134ace63	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin	8300cc17af	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin	6273385618	app/vmagent/remotewrite: properly dial TCP6 addresses set via -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469	2020-05-12 15:22:29 +03:00
Aliaksandr Valialkin	dbd0c552d5	lib/storage: gradually pre-populate per-day inverted index for the next day ... This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:05 +03:00
Roman Khavronenko	8c8ff5d0cb	vmalert: cleanup and restructure of code to improve maintainability (#471) ... The change introduces new entity `manager` which replaces `watchdog`, decouples requestHandler and groups. Manager supposed to control life cycle of groups, rules and config reloads. Groups export an ID method which returns a hash from filename and group name. ID supposed to be unique identifier across all loaded groups. Some tests were added to improve coverage. Bug with wrong annotation value if $value is used in templates after metrics being restored fixed. Notifier interface was extended to accept context. New set of metrics was introduced for config reload.	2020-05-10 17:58:17 +01:00
Nikolay Khramchikhin	9e8733ff65	vmalert config reload ... added config hot reload for vmalert with sighup and api call	2020-05-09 10:32:12 +01:00
Aliaksandr Valialkin	baedb25936	docs/vmauth.md: fix a link to docker images	2020-05-08 14:10:04 +03:00
Aliaksandr Valialkin	51291015a5	app/vmagent: return 200 from /-/reload endpoint as Prometheus does	2020-05-07 19:30:30 +03:00
Aliaksandr Valialkin	6afb25fd08	docs/{vmagent,vmauth}: small clarifications in the docs	2020-05-07 12:55:20 +03:00
Aliaksandr Valialkin	653d51694a	app/vmauth: prevent from attacks with .. in path for accessing resources outside the configured url_prefix	2020-05-07 12:55:18 +03:00
Aliaksandr Valialkin	8a00807f60	app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url	2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin	b69eb7bf38	app/vmagent: properly set client-side TLS certificates for -remoteWrite.url. Previously they were mistakenly set as server-side	2020-05-06 16:50:30 +03:00
Aliaksandr Valialkin	e8936c9cb3	docs/vmagent.md: small fixes	2020-05-06 14:49:18 +03:00
Aliaksandr Valialkin	3f52a97f9b	lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs	2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin	08320cfcf4	docs/{vmauth,vmagent}: fix ports for profiling	2020-05-05 20:15:47 +03:00
Aliaksandr Valialkin	f65930b34d	docs/vmauth.md: mention that we can help creating customized proxy	2020-05-05 12:34:42 +03:00
Aliaksandr Valialkin	266327642b	docs/{vmagent,vmauth}: add Profiling section	2020-05-05 11:45:13 +03:00
Aliaksandr Valialkin	0c7cddfca6	docs: add vmauth.md	2020-05-05 11:17:23 +03:00
Aliaksandr Valialkin	e767aedd17	app/vmauth: add initial version of vmauth. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details	2020-05-05 10:54:17 +03:00
Aliaksandr Valialkin	b5a780930d	docs/vmagent.md: /targets page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead	2020-05-05 10:54:14 +03:00
Roman Khavronenko	0ba1b5c71b	app/vmalert: restore alerts state from datasource metrics (#461) ... * app/vmalert: restore alerts state from datasource metrics Vmalert will restore alerts state for rules that have `rule.For` > 0 from previously written timeseries via `remotewrite.url` flag. * app/vmalert: mention remotewerite and remoteread configuration in README	2020-05-05 00:51:22 +03:00
Aliaksandr Valialkin	40c3ffb359	lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin	432187ac3b	app/vminsert: add /-/reload handler in the same way as for vmagent	2020-04-30 02:15:39 +03:00
DexterZhang	67511d4165	feat(vmagent): add promscrap config reload suppport via http (#450) ... * feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload` * fix: typo fix	2020-04-30 02:00:32 +03:00
Aliaksandr Valialkin	43c39dc36c	vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp ... The upstream fasthttp may contain issues like 996610f021 , plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.	2020-04-29 17:33:34 +03:00
Artem Navoiev	4487b454a8	Update README.md	2020-04-29 12:39:15 +03:00
Aliaksandr Valialkin	57407cca83	app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag ... Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin	4e4f57b121	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin	17d96e4503	app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs vs false-positive not enough memory errors	2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin	1397612117	app/vmalert: added missing comments for public entities	2020-04-28 11:21:07 +03:00
Roman Khavronenko	3bfa41a95c	app/vmalert: initial remote-write support for alerts state persistence. (#442) ... * app/vmalert: initial remote-write support for alerts state persistence. If `remotewrite.url` flag is set, vmalert will send alerts state via remote-write protocol to remote storage. The sending is asynchronous to avoid blocking calls in rules evaluation loop. * app/vmalert: merge with master * app/vmalert: write both `instant` and `for` alerts timeseries states in remote storage.	2020-04-28 00:18:02 +03:00
Aliaksandr Valialkin	90670cb55e	app/vmalert: include it into the next release	2020-04-28 00:10:12 +03:00
Aliaksandr Valialkin	b768bc9a6a	lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs	2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin	b4afe562c1	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin	0224071ebe	lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config	2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin	fcf57f9883	app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks ... This should minimize the number of disk seeks when reading data from temporary file.	2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin	6954d0edb7	lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered	2020-04-26 14:34:11 +03:00
kreedom	2c18548e08	alert - rename validate function and flags (#440) ... * alert - rename validate function and flags	2020-04-26 14:15:04 +03:00
kreedom	5f61d43db9	vmalert - validate template in labels (#439)	2020-04-26 13:53:57 +03:00
肖贝贝	eeadfccdc5	fix: fix vmalert template label not complete bug (#435) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-26 13:30:10 +03:00
Aliaksandr Valialkin	1f3fd93b58	docs/{vmbackup,vmrestore}.md: update -help output	2020-04-24 22:44:21 +03:00
Jason Gardner	66af7e40f3	app/vmbackup: added ability to create and delete snapshots during backup (#428) ... * app/vmbackup: added ability to create and delete snapshots during backup Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422 * Add snapshot create and delete url flags * Fixed errcheck warnings in build	2020-04-24 22:35:03 +03:00
Aliaksandr Valialkin	a596aec82c	app/vmselect: fix description for -search.resetCacheAuthKey	2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin	9ef5935552	lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin	364db13c9c	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin	9ebc937685	app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin	fe57d46687	app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( ... Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin	851946af1e	deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var ... For example, the following command will build VictoriaMetrics docker image on top of alpine image: ROOT_IMAGE=alpine make package-victoria-metrics	2020-04-20 01:16:57 +03:00
Aliaksandr Valialkin	936fb0eac3	app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection ... This should fix the following error when sending data to remote storage: couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-17 15:52:42 +03:00
Aliaksandr Valialkin	79fb595732	docs/vmagent.md: typo fix: unvailable -> unavailable	2020-04-17 13:11:31 +03:00
Aliaksandr Valialkin	546d26523c	app/vmagent/README.md: mention about prodmscrape.suppressScrapeErrors	2020-04-17 13:08:21 +03:00
Aliaksandr Valialkin	f41e6a7bd9	app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query	2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin	071fdf5518	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin	6f7f64f757	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin	067c7afebc	lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page ... This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin	0b76c27fa1	docs/vmagent.md: mention that vmagent supports kubernetes_sd_configs now	2020-04-13 21:06:36 +03:00
hagen1778	2eed6c393f	vmalert: prepare package for external usage ... * update README according to changes * add Makefile with basic commands	2020-04-12 15:32:42 +03:00
kreedom	948f8b6b5f	[vmalert] fix linter issues	2020-04-12 15:08:11 +03:00
kreedom	8fca5f2819	[vmalert] add tests to webserver (#413)	2020-04-12 14:51:03 +03:00